2023-04-12 15:53:32,892 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a77bf0d8-cc1b-e9fb-fa24-4731623e8a8f 2023-04-12 15:53:32,906 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.TestAcidGuaranteesWithEagerPolicy timeout: 13 mins 2023-04-12 15:53:32,922 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=1, rsPorts=, rsClass=null, numDataNodes=1, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-04-12 15:53:32,922 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a77bf0d8-cc1b-e9fb-fa24-4731623e8a8f/cluster_d313ccff-bce6-73f3-294c-14f839051cbd, deleteOnExit=true 2023-04-12 15:53:32,923 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-04-12 15:53:32,923 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a77bf0d8-cc1b-e9fb-fa24-4731623e8a8f/test.cache.data in system properties and HBase conf 2023-04-12 15:53:32,924 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a77bf0d8-cc1b-e9fb-fa24-4731623e8a8f/hadoop.tmp.dir in system properties and HBase conf 2023-04-12 15:53:32,925 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a77bf0d8-cc1b-e9fb-fa24-4731623e8a8f/hadoop.log.dir in system properties and HBase conf 2023-04-12 15:53:32,925 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a77bf0d8-cc1b-e9fb-fa24-4731623e8a8f/mapreduce.cluster.local.dir in system properties and HBase conf 2023-04-12 15:53:32,926 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a77bf0d8-cc1b-e9fb-fa24-4731623e8a8f/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-04-12 15:53:32,926 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-04-12 15:53:33,049 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-04-12 15:53:33,481 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-04-12 15:53:33,487 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a77bf0d8-cc1b-e9fb-fa24-4731623e8a8f/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-04-12 15:53:33,488 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a77bf0d8-cc1b-e9fb-fa24-4731623e8a8f/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-04-12 15:53:33,488 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a77bf0d8-cc1b-e9fb-fa24-4731623e8a8f/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-04-12 15:53:33,489 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a77bf0d8-cc1b-e9fb-fa24-4731623e8a8f/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-12 15:53:33,489 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a77bf0d8-cc1b-e9fb-fa24-4731623e8a8f/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-04-12 15:53:33,490 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a77bf0d8-cc1b-e9fb-fa24-4731623e8a8f/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-04-12 15:53:33,490 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a77bf0d8-cc1b-e9fb-fa24-4731623e8a8f/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-12 15:53:33,490 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a77bf0d8-cc1b-e9fb-fa24-4731623e8a8f/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-12 15:53:33,491 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a77bf0d8-cc1b-e9fb-fa24-4731623e8a8f/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-04-12 15:53:33,491 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a77bf0d8-cc1b-e9fb-fa24-4731623e8a8f/nfs.dump.dir in system properties and HBase conf 2023-04-12 15:53:33,492 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a77bf0d8-cc1b-e9fb-fa24-4731623e8a8f/java.io.tmpdir in system properties and HBase conf 2023-04-12 15:53:33,492 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a77bf0d8-cc1b-e9fb-fa24-4731623e8a8f/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-12 15:53:33,493 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a77bf0d8-cc1b-e9fb-fa24-4731623e8a8f/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-04-12 15:53:33,493 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a77bf0d8-cc1b-e9fb-fa24-4731623e8a8f/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-04-12 15:53:34,102 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-12 15:53:34,106 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-12 15:53:34,382 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-04-12 15:53:34,574 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-04-12 15:53:34,592 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-12 15:53:34,630 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-04-12 15:53:34,663 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a77bf0d8-cc1b-e9fb-fa24-4731623e8a8f/java.io.tmpdir/Jetty_localhost_33487_hdfs____.e6k4g3/webapp 2023-04-12 15:53:34,807 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:33487 2023-04-12 15:53:34,823 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-12 15:53:34,824 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-12 15:53:35,402 WARN [Listener at localhost/40585] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-12 15:53:35,479 WARN [Listener at localhost/40585] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-04-12 15:53:35,503 WARN [Listener at localhost/40585] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-12 15:53:35,511 INFO [Listener at localhost/40585] log.Slf4jLog(67): jetty-6.1.26 2023-04-12 15:53:35,518 INFO [Listener at localhost/40585] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a77bf0d8-cc1b-e9fb-fa24-4731623e8a8f/java.io.tmpdir/Jetty_localhost_38975_datanode____.67v4jr/webapp 2023-04-12 15:53:35,627 INFO [Listener at localhost/40585] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:38975 2023-04-12 15:53:36,010 WARN [Listener at localhost/43583] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-12 15:53:36,578 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x480c526577ff8d4: Processing first storage report for DS-a5fafe52-cee1-4046-a016-6cbf1942c99e from datanode 7b7d27e7-67e1-443a-9646-93169ca9f830 2023-04-12 15:53:36,580 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x480c526577ff8d4: from storage DS-a5fafe52-cee1-4046-a016-6cbf1942c99e node DatanodeRegistration(127.0.0.1:34561, datanodeUuid=7b7d27e7-67e1-443a-9646-93169ca9f830, infoPort=35333, infoSecurePort=0, ipcPort=43583, storageInfo=lv=-57;cid=testClusterID;nsid=310339367;c=1681314814180), blocks: 0, hasStaleStorage: true, processing time: 2 msecs, invalidatedBlocks: 0 2023-04-12 15:53:36,580 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x480c526577ff8d4: Processing first storage report for DS-03d6d56b-ea47-4bfb-bdd4-9a2affd649ac from datanode 7b7d27e7-67e1-443a-9646-93169ca9f830 2023-04-12 15:53:36,580 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x480c526577ff8d4: from storage DS-03d6d56b-ea47-4bfb-bdd4-9a2affd649ac node DatanodeRegistration(127.0.0.1:34561, datanodeUuid=7b7d27e7-67e1-443a-9646-93169ca9f830, infoPort=35333, infoSecurePort=0, ipcPort=43583, storageInfo=lv=-57;cid=testClusterID;nsid=310339367;c=1681314814180), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-04-12 15:53:36,617 DEBUG [Listener at localhost/43583] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a77bf0d8-cc1b-e9fb-fa24-4731623e8a8f 2023-04-12 15:53:36,677 INFO [Listener at localhost/43583] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a77bf0d8-cc1b-e9fb-fa24-4731623e8a8f/cluster_d313ccff-bce6-73f3-294c-14f839051cbd/zookeeper_0, clientPort=60154, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a77bf0d8-cc1b-e9fb-fa24-4731623e8a8f/cluster_d313ccff-bce6-73f3-294c-14f839051cbd/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a77bf0d8-cc1b-e9fb-fa24-4731623e8a8f/cluster_d313ccff-bce6-73f3-294c-14f839051cbd/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-04-12 15:53:36,689 INFO [Listener at localhost/43583] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=60154 2023-04-12 15:53:36,696 INFO [Listener at localhost/43583] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-12 15:53:36,698 INFO [Listener at localhost/43583] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-12 15:53:37,373 INFO [Listener at localhost/43583] util.FSUtils(471): Created version file at hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8 with version=8 2023-04-12 15:53:37,373 INFO [Listener at localhost/43583] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/hbase-staging 2023-04-12 15:53:37,696 INFO [Listener at localhost/43583] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-04-12 15:53:38,178 INFO [Listener at localhost/43583] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=45 2023-04-12 15:53:38,213 INFO [Listener at localhost/43583] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-12 15:53:38,213 INFO [Listener at localhost/43583] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-12 15:53:38,213 INFO [Listener at localhost/43583] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-12 15:53:38,214 INFO [Listener at localhost/43583] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-12 15:53:38,214 INFO [Listener at localhost/43583] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-12 15:53:38,365 INFO [Listener at localhost/43583] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-04-12 15:53:38,437 DEBUG [Listener at localhost/43583] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-04-12 15:53:38,533 INFO [Listener at localhost/43583] ipc.NettyRpcServer(120): Bind to /172.31.14.131:36273 2023-04-12 15:53:38,544 INFO [Listener at localhost/43583] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-12 15:53:38,546 INFO [Listener at localhost/43583] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-12 15:53:38,568 INFO [Listener at localhost/43583] zookeeper.RecoverableZooKeeper(93): Process identifier=master:36273 connecting to ZooKeeper ensemble=127.0.0.1:60154 2023-04-12 15:53:38,610 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:362730x0, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-12 15:53:38,613 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:36273-0x10179140c5c0000 connected 2023-04-12 15:53:38,646 DEBUG [Listener at localhost/43583] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-12 15:53:38,646 DEBUG [Listener at localhost/43583] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-12 15:53:38,650 DEBUG [Listener at localhost/43583] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-12 15:53:38,658 DEBUG [Listener at localhost/43583] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=36273 2023-04-12 15:53:38,658 DEBUG [Listener at localhost/43583] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=36273 2023-04-12 15:53:38,659 DEBUG [Listener at localhost/43583] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=36273 2023-04-12 15:53:38,659 DEBUG [Listener at localhost/43583] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=36273 2023-04-12 15:53:38,659 DEBUG [Listener at localhost/43583] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=36273 2023-04-12 15:53:38,667 INFO [Listener at localhost/43583] master.HMaster(444): hbase.rootdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8, hbase.cluster.distributed=false 2023-04-12 15:53:38,739 INFO [Listener at localhost/43583] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-04-12 15:53:38,740 INFO [Listener at localhost/43583] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-12 15:53:38,740 INFO [Listener at localhost/43583] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-12 15:53:38,740 INFO [Listener at localhost/43583] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-12 15:53:38,740 INFO [Listener at localhost/43583] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-12 15:53:38,740 INFO [Listener at localhost/43583] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-12 15:53:38,745 INFO [Listener at localhost/43583] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-04-12 15:53:38,748 INFO [Listener at localhost/43583] ipc.NettyRpcServer(120): Bind to /172.31.14.131:38905 2023-04-12 15:53:38,750 INFO [Listener at localhost/43583] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-04-12 15:53:38,756 DEBUG [Listener at localhost/43583] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-04-12 15:53:38,757 INFO [Listener at localhost/43583] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-12 15:53:38,760 INFO [Listener at localhost/43583] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-12 15:53:38,762 INFO [Listener at localhost/43583] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:38905 connecting to ZooKeeper ensemble=127.0.0.1:60154 2023-04-12 15:53:38,765 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:389050x0, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-12 15:53:38,767 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:38905-0x10179140c5c0001 connected 2023-04-12 15:53:38,767 DEBUG [Listener at localhost/43583] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-12 15:53:38,768 DEBUG [Listener at localhost/43583] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-12 15:53:38,769 DEBUG [Listener at localhost/43583] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-12 15:53:38,770 DEBUG [Listener at localhost/43583] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=38905 2023-04-12 15:53:38,772 DEBUG [Listener at localhost/43583] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=38905 2023-04-12 15:53:38,775 DEBUG [Listener at localhost/43583] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=38905 2023-04-12 15:53:38,776 DEBUG [Listener at localhost/43583] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=38905 2023-04-12 15:53:38,777 DEBUG [Listener at localhost/43583] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=38905 2023-04-12 15:53:38,780 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,36273,1681314817519 2023-04-12 15:53:38,793 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-12 15:53:38,794 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,36273,1681314817519 2023-04-12 15:53:38,813 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-12 15:53:38,813 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-12 15:53:38,813 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-12 15:53:38,814 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-12 15:53:38,815 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,36273,1681314817519 from backup master directory 2023-04-12 15:53:38,815 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-12 15:53:38,818 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,36273,1681314817519 2023-04-12 15:53:38,819 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-12 15:53:38,819 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-12 15:53:38,819 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase4.apache.org,36273,1681314817519 2023-04-12 15:53:38,822 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-04-12 15:53:38,823 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-04-12 15:53:38,910 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/hbase.id with ID: db2c2c4d-cdd9-40ab-a91c-51592f5bf773 2023-04-12 15:53:38,953 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-12 15:53:38,970 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-12 15:53:39,011 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x7afa6221 to 127.0.0.1:60154 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 15:53:39,051 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@761fd0a5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 15:53:39,079 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-12 15:53:39,081 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-04-12 15:53:39,098 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-04-12 15:53:39,098 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-04-12 15:53:39,100 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-12 15:53:39,103 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-12 15:53:39,105 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-12 15:53:39,140 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/MasterData/data/master/store-tmp 2023-04-12 15:53:39,169 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-12 15:53:39,169 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-12 15:53:39,169 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-12 15:53:39,169 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-12 15:53:39,169 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-12 15:53:39,169 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-12 15:53:39,169 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-12 15:53:39,170 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-12 15:53:39,171 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/MasterData/WALs/jenkins-hbase4.apache.org,36273,1681314817519 2023-04-12 15:53:39,193 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C36273%2C1681314817519, suffix=, logDir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/MasterData/WALs/jenkins-hbase4.apache.org,36273,1681314817519, archiveDir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/MasterData/oldWALs, maxLogs=10 2023-04-12 15:53:39,252 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:34561,DS-a5fafe52-cee1-4046-a016-6cbf1942c99e,DISK] 2023-04-12 15:53:39,260 DEBUG [RS-EventLoopGroup-3-2] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-12 15:53:39,324 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/MasterData/WALs/jenkins-hbase4.apache.org,36273,1681314817519/jenkins-hbase4.apache.org%2C36273%2C1681314817519.1681314819205 2023-04-12 15:53:39,325 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:34561,DS-a5fafe52-cee1-4046-a016-6cbf1942c99e,DISK]] 2023-04-12 15:53:39,326 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-04-12 15:53:39,326 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-12 15:53:39,330 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-04-12 15:53:39,332 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-04-12 15:53:39,411 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-04-12 15:53:39,418 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-04-12 15:53:39,445 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-04-12 15:53:39,457 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 15:53:39,463 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-12 15:53:39,464 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-12 15:53:39,481 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-04-12 15:53:39,486 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-12 15:53:39,487 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59229458, jitterRate=-0.11741229891777039}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-12 15:53:39,487 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-12 15:53:39,491 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-04-12 15:53:39,514 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-04-12 15:53:39,514 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-04-12 15:53:39,517 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-12 15:53:39,518 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-04-12 15:53:39,555 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 36 msec 2023-04-12 15:53:39,555 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-04-12 15:53:39,582 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-04-12 15:53:39,588 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-04-12 15:53:39,622 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-04-12 15:53:39,627 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-04-12 15:53:39,630 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-04-12 15:53:39,637 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-04-12 15:53:39,643 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-04-12 15:53:39,647 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-12 15:53:39,649 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-04-12 15:53:39,650 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-04-12 15:53:39,664 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-04-12 15:53:39,669 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-12 15:53:39,669 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-12 15:53:39,669 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-12 15:53:39,670 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase4.apache.org,36273,1681314817519, sessionid=0x10179140c5c0000, setting cluster-up flag (Was=false) 2023-04-12 15:53:39,689 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-12 15:53:39,695 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-04-12 15:53:39,697 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,36273,1681314817519 2023-04-12 15:53:39,703 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-12 15:53:39,709 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-04-12 15:53:39,711 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,36273,1681314817519 2023-04-12 15:53:39,713 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.hbase-snapshot/.tmp 2023-04-12 15:53:39,781 INFO [RS:0;jenkins-hbase4:38905] regionserver.HRegionServer(952): ClusterId : db2c2c4d-cdd9-40ab-a91c-51592f5bf773 2023-04-12 15:53:39,786 DEBUG [RS:0;jenkins-hbase4:38905] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-04-12 15:53:39,794 DEBUG [RS:0;jenkins-hbase4:38905] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-04-12 15:53:39,795 DEBUG [RS:0;jenkins-hbase4:38905] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-04-12 15:53:39,801 DEBUG [RS:0;jenkins-hbase4:38905] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-04-12 15:53:39,802 DEBUG [RS:0;jenkins-hbase4:38905] zookeeper.ReadOnlyZKClient(139): Connect 0x0d7a4eb8 to 127.0.0.1:60154 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 15:53:39,806 DEBUG [RS:0;jenkins-hbase4:38905] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3bdb167, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 15:53:39,807 DEBUG [RS:0;jenkins-hbase4:38905] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@78225c8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-04-12 15:53:39,840 DEBUG [RS:0;jenkins-hbase4:38905] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase4:38905 2023-04-12 15:53:39,843 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-04-12 15:53:39,843 INFO [RS:0;jenkins-hbase4:38905] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-04-12 15:53:39,844 INFO [RS:0;jenkins-hbase4:38905] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-04-12 15:53:39,844 DEBUG [RS:0;jenkins-hbase4:38905] regionserver.HRegionServer(1023): About to register with Master. 2023-04-12 15:53:39,846 INFO [RS:0;jenkins-hbase4:38905] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,36273,1681314817519 with isa=jenkins-hbase4.apache.org/172.31.14.131:38905, startcode=1681314818739 2023-04-12 15:53:39,853 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-12 15:53:39,853 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-12 15:53:39,853 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-12 15:53:39,853 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-12 15:53:39,853 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-04-12 15:53:39,853 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-12 15:53:39,853 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-04-12 15:53:39,853 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-12 15:53:39,855 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1681314849854 2023-04-12 15:53:39,856 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-04-12 15:53:39,861 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-04-12 15:53:39,861 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-04-12 15:53:39,864 DEBUG [RS:0;jenkins-hbase4:38905] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-04-12 15:53:39,866 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-12 15:53:39,868 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-04-12 15:53:39,877 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-04-12 15:53:39,877 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-04-12 15:53:39,878 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-04-12 15:53:39,878 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-04-12 15:53:39,880 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-12 15:53:39,881 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-04-12 15:53:39,883 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-04-12 15:53:39,883 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-04-12 15:53:39,887 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-04-12 15:53:39,887 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-04-12 15:53:39,889 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1681314819889,5,FailOnTimeoutGroup] 2023-04-12 15:53:39,891 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1681314819890,5,FailOnTimeoutGroup] 2023-04-12 15:53:39,891 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-12 15:53:39,891 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-04-12 15:53:39,892 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-04-12 15:53:39,892 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-04-12 15:53:39,920 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39659, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-04-12 15:53:39,931 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:39,948 DEBUG [RS:0;jenkins-hbase4:38905] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8 2023-04-12 15:53:39,948 DEBUG [RS:0;jenkins-hbase4:38905] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:40585 2023-04-12 15:53:39,948 DEBUG [RS:0;jenkins-hbase4:38905] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-04-12 15:53:39,953 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-12 15:53:39,953 DEBUG [RS:0;jenkins-hbase4:38905] zookeeper.ZKUtil(162): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:39,953 WARN [RS:0;jenkins-hbase4:38905] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-12 15:53:39,954 INFO [RS:0;jenkins-hbase4:38905] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-12 15:53:39,954 DEBUG [RS:0;jenkins-hbase4:38905] regionserver.HRegionServer(1947): logDir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/WALs/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:39,957 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,38905,1681314818739] 2023-04-12 15:53:39,965 DEBUG [RS:0;jenkins-hbase4:38905] zookeeper.ZKUtil(162): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:39,977 DEBUG [RS:0;jenkins-hbase4:38905] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-04-12 15:53:39,988 INFO [RS:0;jenkins-hbase4:38905] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-04-12 15:53:40,010 INFO [RS:0;jenkins-hbase4:38905] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-04-12 15:53:40,014 INFO [RS:0;jenkins-hbase4:38905] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-04-12 15:53:40,014 INFO [RS:0;jenkins-hbase4:38905] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-04-12 15:53:40,016 INFO [RS:0;jenkins-hbase4:38905] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-04-12 15:53:40,024 INFO [RS:0;jenkins-hbase4:38905] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-04-12 15:53:40,024 DEBUG [RS:0;jenkins-hbase4:38905] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-12 15:53:40,024 DEBUG [RS:0;jenkins-hbase4:38905] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-12 15:53:40,025 DEBUG [RS:0;jenkins-hbase4:38905] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-12 15:53:40,025 DEBUG [RS:0;jenkins-hbase4:38905] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-12 15:53:40,025 DEBUG [RS:0;jenkins-hbase4:38905] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-12 15:53:40,025 DEBUG [RS:0;jenkins-hbase4:38905] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-04-12 15:53:40,025 DEBUG [RS:0;jenkins-hbase4:38905] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-12 15:53:40,025 DEBUG [RS:0;jenkins-hbase4:38905] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-12 15:53:40,025 DEBUG [RS:0;jenkins-hbase4:38905] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-12 15:53:40,026 DEBUG [RS:0;jenkins-hbase4:38905] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-12 15:53:40,026 INFO [RS:0;jenkins-hbase4:38905] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-04-12 15:53:40,027 INFO [RS:0;jenkins-hbase4:38905] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-04-12 15:53:40,027 INFO [RS:0;jenkins-hbase4:38905] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-04-12 15:53:40,045 INFO [RS:0;jenkins-hbase4:38905] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-04-12 15:53:40,047 INFO [RS:0;jenkins-hbase4:38905] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,38905,1681314818739-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-12 15:53:40,063 INFO [RS:0;jenkins-hbase4:38905] regionserver.Replication(203): jenkins-hbase4.apache.org,38905,1681314818739 started 2023-04-12 15:53:40,063 INFO [RS:0;jenkins-hbase4:38905] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,38905,1681314818739, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:38905, sessionid=0x10179140c5c0001 2023-04-12 15:53:40,063 DEBUG [RS:0;jenkins-hbase4:38905] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-04-12 15:53:40,064 DEBUG [RS:0;jenkins-hbase4:38905] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:40,064 DEBUG [RS:0;jenkins-hbase4:38905] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,38905,1681314818739' 2023-04-12 15:53:40,064 DEBUG [RS:0;jenkins-hbase4:38905] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:53:40,064 DEBUG [RS:0;jenkins-hbase4:38905] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:53:40,065 DEBUG [RS:0;jenkins-hbase4:38905] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-04-12 15:53:40,065 DEBUG [RS:0;jenkins-hbase4:38905] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-04-12 15:53:40,065 DEBUG [RS:0;jenkins-hbase4:38905] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:40,065 DEBUG [RS:0;jenkins-hbase4:38905] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,38905,1681314818739' 2023-04-12 15:53:40,065 DEBUG [RS:0;jenkins-hbase4:38905] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-04-12 15:53:40,066 DEBUG [RS:0;jenkins-hbase4:38905] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-04-12 15:53:40,066 DEBUG [RS:0;jenkins-hbase4:38905] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-04-12 15:53:40,066 INFO [RS:0;jenkins-hbase4:38905] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-04-12 15:53:40,066 INFO [RS:0;jenkins-hbase4:38905] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-04-12 15:53:40,178 INFO [RS:0;jenkins-hbase4:38905] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C38905%2C1681314818739, suffix=, logDir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/WALs/jenkins-hbase4.apache.org,38905,1681314818739, archiveDir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/oldWALs, maxLogs=32 2023-04-12 15:53:40,200 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:34561,DS-a5fafe52-cee1-4046-a016-6cbf1942c99e,DISK] 2023-04-12 15:53:40,209 INFO [RS:0;jenkins-hbase4:38905] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/WALs/jenkins-hbase4.apache.org,38905,1681314818739/jenkins-hbase4.apache.org%2C38905%2C1681314818739.1681314820181 2023-04-12 15:53:40,211 DEBUG [RS:0;jenkins-hbase4:38905] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:34561,DS-a5fafe52-cee1-4046-a016-6cbf1942c99e,DISK]] 2023-04-12 15:53:40,310 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-12 15:53:40,311 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-12 15:53:40,312 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8 2023-04-12 15:53:40,734 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-12 15:53:40,737 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-12 15:53:40,740 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/hbase/meta/1588230740/info 2023-04-12 15:53:40,741 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-12 15:53:40,743 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 15:53:40,743 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-12 15:53:40,745 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/hbase/meta/1588230740/rep_barrier 2023-04-12 15:53:40,746 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-12 15:53:40,747 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 15:53:40,747 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-12 15:53:40,749 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/hbase/meta/1588230740/table 2023-04-12 15:53:40,751 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-12 15:53:40,752 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 15:53:40,754 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/hbase/meta/1588230740 2023-04-12 15:53:40,755 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/hbase/meta/1588230740 2023-04-12 15:53:40,759 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-12 15:53:40,760 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-12 15:53:40,764 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-12 15:53:40,765 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71189912, jitterRate=0.06081235408782959}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-12 15:53:40,765 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-12 15:53:40,765 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-12 15:53:40,765 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-12 15:53:40,766 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-12 15:53:40,766 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-04-12 15:53:40,766 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-12 15:53:40,766 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-12 15:53:40,767 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-12 15:53:40,772 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-04-12 15:53:40,772 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-04-12 15:53:40,782 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-04-12 15:53:40,795 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-04-12 15:53:40,798 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-04-12 15:53:40,950 DEBUG [jenkins-hbase4:36273] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-04-12 15:53:40,953 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,38905,1681314818739, state=OPENING 2023-04-12 15:53:40,962 DEBUG [PEWorker-3] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-04-12 15:53:40,963 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-12 15:53:40,964 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-12 15:53:40,966 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,38905,1681314818739}] 2023-04-12 15:53:41,152 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:41,155 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-04-12 15:53:41,159 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33744, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-04-12 15:53:41,174 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-04-12 15:53:41,174 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-12 15:53:41,178 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C38905%2C1681314818739.meta, suffix=.meta, logDir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/WALs/jenkins-hbase4.apache.org,38905,1681314818739, archiveDir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/oldWALs, maxLogs=32 2023-04-12 15:53:41,196 DEBUG [RS-EventLoopGroup-3-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:34561,DS-a5fafe52-cee1-4046-a016-6cbf1942c99e,DISK] 2023-04-12 15:53:41,201 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/WALs/jenkins-hbase4.apache.org,38905,1681314818739/jenkins-hbase4.apache.org%2C38905%2C1681314818739.meta.1681314821179.meta 2023-04-12 15:53:41,203 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:34561,DS-a5fafe52-cee1-4046-a016-6cbf1942c99e,DISK]] 2023-04-12 15:53:41,203 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-04-12 15:53:41,205 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-04-12 15:53:41,227 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-04-12 15:53:41,233 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-04-12 15:53:41,239 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-04-12 15:53:41,239 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-12 15:53:41,239 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-04-12 15:53:41,239 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-04-12 15:53:41,242 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-12 15:53:41,244 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/hbase/meta/1588230740/info 2023-04-12 15:53:41,244 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/hbase/meta/1588230740/info 2023-04-12 15:53:41,245 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-12 15:53:41,246 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 15:53:41,246 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-12 15:53:41,247 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/hbase/meta/1588230740/rep_barrier 2023-04-12 15:53:41,247 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/hbase/meta/1588230740/rep_barrier 2023-04-12 15:53:41,248 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-12 15:53:41,249 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 15:53:41,249 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-12 15:53:41,251 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/hbase/meta/1588230740/table 2023-04-12 15:53:41,251 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/hbase/meta/1588230740/table 2023-04-12 15:53:41,252 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-12 15:53:41,252 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 15:53:41,254 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/hbase/meta/1588230740 2023-04-12 15:53:41,257 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/hbase/meta/1588230740 2023-04-12 15:53:41,260 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-12 15:53:41,262 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-12 15:53:41,263 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65313919, jitterRate=-0.02674676477909088}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-12 15:53:41,264 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-12 15:53:41,273 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1681314821144 2023-04-12 15:53:41,292 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-04-12 15:53:41,293 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-04-12 15:53:41,293 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,38905,1681314818739, state=OPEN 2023-04-12 15:53:41,296 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-04-12 15:53:41,296 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-12 15:53:41,300 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-04-12 15:53:41,300 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,38905,1681314818739 in 330 msec 2023-04-12 15:53:41,304 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-04-12 15:53:41,305 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 519 msec 2023-04-12 15:53:41,309 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 1.5420 sec 2023-04-12 15:53:41,310 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1681314821309, completionTime=-1 2023-04-12 15:53:41,310 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-04-12 15:53:41,310 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-04-12 15:53:41,371 DEBUG [hconnection-0x5279ba2c-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 15:53:41,375 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33758, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 15:53:41,394 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-04-12 15:53:41,395 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1681314881395 2023-04-12 15:53:41,395 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1681314941395 2023-04-12 15:53:41,395 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 84 msec 2023-04-12 15:53:41,420 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,36273,1681314817519-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-12 15:53:41,420 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,36273,1681314817519-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-12 15:53:41,420 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,36273,1681314817519-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-12 15:53:41,422 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:36273, period=300000, unit=MILLISECONDS is enabled. 2023-04-12 15:53:41,422 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-04-12 15:53:41,428 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-04-12 15:53:41,436 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-04-12 15:53:41,437 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-12 15:53:41,446 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-04-12 15:53:41,448 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-04-12 15:53:41,450 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-12 15:53:41,470 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/hbase/namespace/1b2138e70df333d56ab86f7247aa4167 2023-04-12 15:53:41,472 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/hbase/namespace/1b2138e70df333d56ab86f7247aa4167 empty. 2023-04-12 15:53:41,473 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/hbase/namespace/1b2138e70df333d56ab86f7247aa4167 2023-04-12 15:53:41,473 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-04-12 15:53:41,496 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-04-12 15:53:41,498 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => 1b2138e70df333d56ab86f7247aa4167, NAME => 'hbase:namespace,,1681314821437.1b2138e70df333d56ab86f7247aa4167.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp 2023-04-12 15:53:41,910 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1681314821437.1b2138e70df333d56ab86f7247aa4167.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-12 15:53:41,910 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing 1b2138e70df333d56ab86f7247aa4167, disabling compactions & flushes 2023-04-12 15:53:41,910 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1681314821437.1b2138e70df333d56ab86f7247aa4167. 2023-04-12 15:53:41,910 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1681314821437.1b2138e70df333d56ab86f7247aa4167. 2023-04-12 15:53:41,911 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1681314821437.1b2138e70df333d56ab86f7247aa4167. after waiting 0 ms 2023-04-12 15:53:41,911 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1681314821437.1b2138e70df333d56ab86f7247aa4167. 2023-04-12 15:53:41,911 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1681314821437.1b2138e70df333d56ab86f7247aa4167. 2023-04-12 15:53:41,911 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for 1b2138e70df333d56ab86f7247aa4167: 2023-04-12 15:53:41,916 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-04-12 15:53:41,932 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1681314821437.1b2138e70df333d56ab86f7247aa4167.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1681314821919"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681314821919"}]},"ts":"1681314821919"} 2023-04-12 15:53:41,966 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-12 15:53:41,968 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-12 15:53:41,972 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681314821968"}]},"ts":"1681314821968"} 2023-04-12 15:53:41,976 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-04-12 15:53:41,985 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=1b2138e70df333d56ab86f7247aa4167, ASSIGN}] 2023-04-12 15:53:41,989 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=1b2138e70df333d56ab86f7247aa4167, ASSIGN 2023-04-12 15:53:41,990 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=1b2138e70df333d56ab86f7247aa4167, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,38905,1681314818739; forceNewPlan=false, retain=false 2023-04-12 15:53:42,141 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=1b2138e70df333d56ab86f7247aa4167, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:42,142 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1681314821437.1b2138e70df333d56ab86f7247aa4167.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1681314822141"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681314822141"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681314822141"}]},"ts":"1681314822141"} 2023-04-12 15:53:42,146 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure 1b2138e70df333d56ab86f7247aa4167, server=jenkins-hbase4.apache.org,38905,1681314818739}] 2023-04-12 15:53:42,305 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1681314821437.1b2138e70df333d56ab86f7247aa4167. 2023-04-12 15:53:42,306 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1b2138e70df333d56ab86f7247aa4167, NAME => 'hbase:namespace,,1681314821437.1b2138e70df333d56ab86f7247aa4167.', STARTKEY => '', ENDKEY => ''} 2023-04-12 15:53:42,306 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 1b2138e70df333d56ab86f7247aa4167 2023-04-12 15:53:42,307 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1681314821437.1b2138e70df333d56ab86f7247aa4167.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-12 15:53:42,307 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1b2138e70df333d56ab86f7247aa4167 2023-04-12 15:53:42,307 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1b2138e70df333d56ab86f7247aa4167 2023-04-12 15:53:42,309 INFO [StoreOpener-1b2138e70df333d56ab86f7247aa4167-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1b2138e70df333d56ab86f7247aa4167 2023-04-12 15:53:42,311 DEBUG [StoreOpener-1b2138e70df333d56ab86f7247aa4167-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/hbase/namespace/1b2138e70df333d56ab86f7247aa4167/info 2023-04-12 15:53:42,311 DEBUG [StoreOpener-1b2138e70df333d56ab86f7247aa4167-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/hbase/namespace/1b2138e70df333d56ab86f7247aa4167/info 2023-04-12 15:53:42,312 INFO [StoreOpener-1b2138e70df333d56ab86f7247aa4167-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1b2138e70df333d56ab86f7247aa4167 columnFamilyName info 2023-04-12 15:53:42,312 INFO [StoreOpener-1b2138e70df333d56ab86f7247aa4167-1] regionserver.HStore(310): Store=1b2138e70df333d56ab86f7247aa4167/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 15:53:42,314 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/hbase/namespace/1b2138e70df333d56ab86f7247aa4167 2023-04-12 15:53:42,315 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/hbase/namespace/1b2138e70df333d56ab86f7247aa4167 2023-04-12 15:53:42,321 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1b2138e70df333d56ab86f7247aa4167 2023-04-12 15:53:42,325 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/hbase/namespace/1b2138e70df333d56ab86f7247aa4167/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-12 15:53:42,326 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1b2138e70df333d56ab86f7247aa4167; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59266687, jitterRate=-0.11685754358768463}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-12 15:53:42,326 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1b2138e70df333d56ab86f7247aa4167: 2023-04-12 15:53:42,328 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1681314821437.1b2138e70df333d56ab86f7247aa4167., pid=6, masterSystemTime=1681314822299 2023-04-12 15:53:42,332 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1681314821437.1b2138e70df333d56ab86f7247aa4167. 2023-04-12 15:53:42,332 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1681314821437.1b2138e70df333d56ab86f7247aa4167. 2023-04-12 15:53:42,333 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=1b2138e70df333d56ab86f7247aa4167, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:42,334 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1681314821437.1b2138e70df333d56ab86f7247aa4167.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1681314822333"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681314822333"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681314822333"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681314822333"}]},"ts":"1681314822333"} 2023-04-12 15:53:42,340 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-04-12 15:53:42,340 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure 1b2138e70df333d56ab86f7247aa4167, server=jenkins-hbase4.apache.org,38905,1681314818739 in 191 msec 2023-04-12 15:53:42,343 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-04-12 15:53:42,344 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=1b2138e70df333d56ab86f7247aa4167, ASSIGN in 355 msec 2023-04-12 15:53:42,345 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-12 15:53:42,346 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681314822345"}]},"ts":"1681314822345"} 2023-04-12 15:53:42,349 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-04-12 15:53:42,354 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-04-12 15:53:42,354 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-04-12 15:53:42,356 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-04-12 15:53:42,356 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-12 15:53:42,358 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 917 msec 2023-04-12 15:53:42,396 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-04-12 15:53:42,414 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-12 15:53:42,424 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 35 msec 2023-04-12 15:53:42,431 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-04-12 15:53:42,442 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-12 15:53:42,449 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 18 msec 2023-04-12 15:53:42,456 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-04-12 15:53:42,459 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-04-12 15:53:42,459 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 3.639sec 2023-04-12 15:53:42,461 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-04-12 15:53:42,463 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-04-12 15:53:42,463 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-04-12 15:53:42,464 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,36273,1681314817519-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-04-12 15:53:42,464 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,36273,1681314817519-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-04-12 15:53:42,474 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-04-12 15:53:42,487 DEBUG [Listener at localhost/43583] zookeeper.ReadOnlyZKClient(139): Connect 0x5e324982 to 127.0.0.1:60154 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 15:53:42,491 DEBUG [Listener at localhost/43583] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@373f9c99, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 15:53:42,504 DEBUG [hconnection-0x258148f-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 15:53:42,514 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33762, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 15:53:42,524 INFO [Listener at localhost/43583] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase4.apache.org,36273,1681314817519 2023-04-12 15:53:42,544 INFO [Listener at localhost/43583] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=207, OpenFileDescriptor=470, MaxFileDescriptor=60000, SystemLoadAverage=354, ProcessCount=169, AvailableMemoryMB=1263 2023-04-12 15:53:42,559 DEBUG [Listener at localhost/43583] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-12 15:53:42,562 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39180, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-12 15:53:42,571 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-12 15:53:42,575 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-12 15:53:42,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-12 15:53:42,579 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-12 15:53:42,581 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-12 15:53:42,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 9 2023-04-12 15:53:42,584 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20 2023-04-12 15:53:42,585 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20 empty. 2023-04-12 15:53:42,587 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20 2023-04-12 15:53:42,587 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-12 15:53:42,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-12 15:53:42,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-12 15:53:42,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-12 15:53:43,003 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-12 15:53:43,005 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => feda8274e3aaad09f0f3a1336e663d20, NAME => 'TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp 2023-04-12 15:53:43,017 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-12 15:53:43,017 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing feda8274e3aaad09f0f3a1336e663d20, disabling compactions & flushes 2023-04-12 15:53:43,017 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:43,017 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:43,017 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. after waiting 0 ms 2023-04-12 15:53:43,017 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:43,018 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:43,018 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for feda8274e3aaad09f0f3a1336e663d20: 2023-04-12 15:53:43,022 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-12 15:53:43,024 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681314823024"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681314823024"}]},"ts":"1681314823024"} 2023-04-12 15:53:43,027 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-12 15:53:43,028 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-12 15:53:43,028 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681314823028"}]},"ts":"1681314823028"} 2023-04-12 15:53:43,031 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-12 15:53:43,035 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=feda8274e3aaad09f0f3a1336e663d20, ASSIGN}] 2023-04-12 15:53:43,037 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=feda8274e3aaad09f0f3a1336e663d20, ASSIGN 2023-04-12 15:53:43,039 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=feda8274e3aaad09f0f3a1336e663d20, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,38905,1681314818739; forceNewPlan=false, retain=false 2023-04-12 15:53:43,190 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=feda8274e3aaad09f0f3a1336e663d20, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:43,191 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681314823190"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681314823190"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681314823190"}]},"ts":"1681314823190"} 2023-04-12 15:53:43,194 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739}] 2023-04-12 15:53:43,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-12 15:53:43,354 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:43,354 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => feda8274e3aaad09f0f3a1336e663d20, NAME => 'TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.', STARTKEY => '', ENDKEY => ''} 2023-04-12 15:53:43,354 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees feda8274e3aaad09f0f3a1336e663d20 2023-04-12 15:53:43,355 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-12 15:53:43,355 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for feda8274e3aaad09f0f3a1336e663d20 2023-04-12 15:53:43,355 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for feda8274e3aaad09f0f3a1336e663d20 2023-04-12 15:53:43,357 INFO [StoreOpener-feda8274e3aaad09f0f3a1336e663d20-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region feda8274e3aaad09f0f3a1336e663d20 2023-04-12 15:53:43,359 DEBUG [StoreOpener-feda8274e3aaad09f0f3a1336e663d20-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A 2023-04-12 15:53:43,359 DEBUG [StoreOpener-feda8274e3aaad09f0f3a1336e663d20-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A 2023-04-12 15:53:43,361 INFO [StoreOpener-feda8274e3aaad09f0f3a1336e663d20-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-12 15:53:43,361 INFO [StoreOpener-feda8274e3aaad09f0f3a1336e663d20-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region feda8274e3aaad09f0f3a1336e663d20 columnFamilyName A 2023-04-12 15:53:43,362 INFO [StoreOpener-feda8274e3aaad09f0f3a1336e663d20-1] regionserver.HStore(310): Store=feda8274e3aaad09f0f3a1336e663d20/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 15:53:43,363 INFO [StoreOpener-feda8274e3aaad09f0f3a1336e663d20-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region feda8274e3aaad09f0f3a1336e663d20 2023-04-12 15:53:43,365 DEBUG [StoreOpener-feda8274e3aaad09f0f3a1336e663d20-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B 2023-04-12 15:53:43,365 DEBUG [StoreOpener-feda8274e3aaad09f0f3a1336e663d20-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B 2023-04-12 15:53:43,365 INFO [StoreOpener-feda8274e3aaad09f0f3a1336e663d20-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-12 15:53:43,365 INFO [StoreOpener-feda8274e3aaad09f0f3a1336e663d20-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region feda8274e3aaad09f0f3a1336e663d20 columnFamilyName B 2023-04-12 15:53:43,366 INFO [StoreOpener-feda8274e3aaad09f0f3a1336e663d20-1] regionserver.HStore(310): Store=feda8274e3aaad09f0f3a1336e663d20/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 15:53:43,366 INFO [StoreOpener-feda8274e3aaad09f0f3a1336e663d20-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region feda8274e3aaad09f0f3a1336e663d20 2023-04-12 15:53:43,369 DEBUG [StoreOpener-feda8274e3aaad09f0f3a1336e663d20-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C 2023-04-12 15:53:43,369 DEBUG [StoreOpener-feda8274e3aaad09f0f3a1336e663d20-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C 2023-04-12 15:53:43,370 INFO [StoreOpener-feda8274e3aaad09f0f3a1336e663d20-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-12 15:53:43,370 INFO [StoreOpener-feda8274e3aaad09f0f3a1336e663d20-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region feda8274e3aaad09f0f3a1336e663d20 columnFamilyName C 2023-04-12 15:53:43,371 INFO [StoreOpener-feda8274e3aaad09f0f3a1336e663d20-1] regionserver.HStore(310): Store=feda8274e3aaad09f0f3a1336e663d20/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 15:53:43,372 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:43,374 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20 2023-04-12 15:53:43,375 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20 2023-04-12 15:53:43,378 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-12 15:53:43,381 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for feda8274e3aaad09f0f3a1336e663d20 2023-04-12 15:53:43,385 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-12 15:53:43,386 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened feda8274e3aaad09f0f3a1336e663d20; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71081314, jitterRate=0.05919411778450012}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-12 15:53:43,386 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for feda8274e3aaad09f0f3a1336e663d20: 2023-04-12 15:53:43,388 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20., pid=11, masterSystemTime=1681314823347 2023-04-12 15:53:43,390 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:43,391 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:43,392 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=feda8274e3aaad09f0f3a1336e663d20, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:43,392 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681314823391"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681314823391"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681314823391"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681314823391"}]},"ts":"1681314823391"} 2023-04-12 15:53:43,400 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-04-12 15:53:43,400 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 in 201 msec 2023-04-12 15:53:43,404 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-04-12 15:53:43,405 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=feda8274e3aaad09f0f3a1336e663d20, ASSIGN in 365 msec 2023-04-12 15:53:43,406 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-12 15:53:43,406 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681314823406"}]},"ts":"1681314823406"} 2023-04-12 15:53:43,409 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-12 15:53:43,414 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-12 15:53:43,417 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 839 msec 2023-04-12 15:53:43,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-12 15:53:43,706 INFO [Listener at localhost/43583] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 9 completed 2023-04-12 15:53:43,711 DEBUG [Listener at localhost/43583] zookeeper.ReadOnlyZKClient(139): Connect 0x35594b1b to 127.0.0.1:60154 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 15:53:43,716 DEBUG [Listener at localhost/43583] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@55e5fd59, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 15:53:43,718 DEBUG [Listener at localhost/43583] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 15:53:43,722 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58330, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 15:53:43,725 DEBUG [Listener at localhost/43583] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-12 15:53:43,728 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46536, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-12 15:53:43,737 DEBUG [Listener at localhost/43583] zookeeper.ReadOnlyZKClient(139): Connect 0x0e16d41b to 127.0.0.1:60154 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 15:53:43,741 DEBUG [Listener at localhost/43583] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@56223d42, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 15:53:43,742 DEBUG [Listener at localhost/43583] zookeeper.ReadOnlyZKClient(139): Connect 0x4fea27db to 127.0.0.1:60154 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 15:53:43,747 DEBUG [Listener at localhost/43583] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@30eb7fb2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 15:53:43,748 DEBUG [Listener at localhost/43583] zookeeper.ReadOnlyZKClient(139): Connect 0x7e728705 to 127.0.0.1:60154 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 15:53:43,754 DEBUG [Listener at localhost/43583] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2de4d1f6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 15:53:43,754 DEBUG [Listener at localhost/43583] zookeeper.ReadOnlyZKClient(139): Connect 0x63859c62 to 127.0.0.1:60154 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 15:53:43,758 DEBUG [Listener at localhost/43583] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2c57e1b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 15:53:43,758 DEBUG [Listener at localhost/43583] zookeeper.ReadOnlyZKClient(139): Connect 0x42927198 to 127.0.0.1:60154 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 15:53:43,762 DEBUG [Listener at localhost/43583] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@60b8c4c7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 15:53:43,763 DEBUG [Listener at localhost/43583] zookeeper.ReadOnlyZKClient(139): Connect 0x04fb2b82 to 127.0.0.1:60154 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 15:53:43,770 DEBUG [Listener at localhost/43583] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@59d6b6ea, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 15:53:43,770 DEBUG [Listener at localhost/43583] zookeeper.ReadOnlyZKClient(139): Connect 0x3cd3abbd to 127.0.0.1:60154 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 15:53:43,806 DEBUG [Listener at localhost/43583] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6129209e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 15:53:43,807 DEBUG [Listener at localhost/43583] zookeeper.ReadOnlyZKClient(139): Connect 0x4b02eeb3 to 127.0.0.1:60154 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 15:53:43,812 DEBUG [Listener at localhost/43583] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@63871a9d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 15:53:43,812 DEBUG [Listener at localhost/43583] zookeeper.ReadOnlyZKClient(139): Connect 0x4808c6d5 to 127.0.0.1:60154 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 15:53:43,816 DEBUG [Listener at localhost/43583] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1423ce11, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 15:53:43,823 DEBUG [hconnection-0x37e8943a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 15:53:43,823 DEBUG [hconnection-0x527b0319-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 15:53:43,824 DEBUG [hconnection-0x544eb923-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 15:53:43,825 DEBUG [hconnection-0x6d40587f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 15:53:43,826 DEBUG [hconnection-0x2b3986b1-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 15:53:43,826 DEBUG [hconnection-0x74b74e95-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 15:53:43,833 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58350, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 15:53:43,836 DEBUG [hconnection-0x55d4bc5a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 15:53:43,837 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58340, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 15:53:43,840 DEBUG [hconnection-0x7576e4ed-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 15:53:43,843 DEBUG [hconnection-0x62a93ad6-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 15:53:43,847 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58352, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 15:53:43,849 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58364, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 15:53:43,852 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58398, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 15:53:43,855 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58388, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 15:53:43,855 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58354, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 15:53:43,857 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58374, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 15:53:43,858 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58414, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 15:53:43,864 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:53:43,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:53:43,893 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:53:43,894 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:53:43,895 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:53:43,895 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:53:43,895 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:43,895 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:43,897 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:53:43,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:53:43,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:53:43,899 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:43,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:43,900 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:43,900 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:53:43,900 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:43,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:53:43,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:43,901 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:53:43,903 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:53:43,905 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:53:43,905 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:53:43,906 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:53:43,923 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:53:43,923 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:53:43,928 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:43,928 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. started... 2023-04-12 15:53:43,929 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing feda8274e3aaad09f0f3a1336e663d20 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-12 15:53:43,953 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK feda8274e3aaad09f0f3a1336e663d20, store=A 2023-04-12 15:53:43,953 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:53:43,954 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK feda8274e3aaad09f0f3a1336e663d20, store=B 2023-04-12 15:53:43,954 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:53:43,954 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK feda8274e3aaad09f0f3a1336e663d20, store=C 2023-04-12 15:53:43,954 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:53:43,979 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. as already flushing 2023-04-12 15:53:43,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on feda8274e3aaad09f0f3a1336e663d20 2023-04-12 15:53:44,073 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=8 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/A/e029725a1c65412ebf79fbcaab14f2c7 2023-04-12 15:53:44,099 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681314824012, max timestamp=1681314824075Num uniques 859; 2023-04-12 15:53:44,099 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681314824012, max timestamp=1681314824075Num uniques 859; 2023-04-12 15:53:44,099 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-12 15:53:44,100 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681314824012, max timestamp=1681314824075Num uniques 859; 2023-04-12 15:53:44,099 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-12 15:53:44,100 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-12 15:53:44,165 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681314824112, max timestamp=1681314824154Num uniques 1018; 2023-04-12 15:53:44,166 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-12 15:53:44,179 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681314824112, max timestamp=1681314824156Num uniques 1009; 2023-04-12 15:53:44,179 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-04-12 15:53:44,196 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681314824112, max timestamp=1681314824154Num uniques 1009; 2023-04-12 15:53:44,205 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-04-12 15:53:44,273 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=233, cellSize=10668, totalHeapSize=24472, min timestamp=1681314824193, max timestamp=1681314824219Num uniques 1027; 2023-04-12 15:53:44,274 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10668, suffix heap size=134776,new segment heap size=24472 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=233 2023-04-12 15:53:44,281 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681314824202, max timestamp=1681314824219Num uniques 1009; 2023-04-12 15:53:44,281 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-04-12 15:53:44,285 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681314824202, max timestamp=1681314824219Num uniques 1009; 2023-04-12 15:53:44,285 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-04-12 15:53:44,341 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:44,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58414 deadline: 1681314884319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:44,344 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:44,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58374 deadline: 1681314884322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:44,352 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:44,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58398 deadline: 1681314884342, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:44,354 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:44,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58340 deadline: 1681314884347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:44,355 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:44,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58354 deadline: 1681314884347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:44,358 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681314824297, max timestamp=1681314824329Num uniques 1009; 2023-04-12 15:53:44,358 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681314824297, max timestamp=1681314824325Num uniques 1009; 2023-04-12 15:53:44,358 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-04-12 15:53:44,359 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-04-12 15:53:44,363 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681314824297, max timestamp=1681314824320Num uniques 1092; 2023-04-12 15:53:44,363 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50012, new segment data size=6870, suffix heap size=141536,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1092, new segment cells count=150 2023-04-12 15:53:44,572 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1681314824562, max timestamp=1681314824567Num uniques 1009; 2023-04-12 15:53:44,572 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1681314824562, max timestamp=1681314824567Num uniques 1009; 2023-04-12 15:53:44,572 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=11220, suffix heap size=132904,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=245 2023-04-12 15:53:44,572 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=11220, suffix heap size=132904,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=245 2023-04-12 15:53:44,573 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1681314824562, max timestamp=1681314824567Num uniques 1009; 2023-04-12 15:53:44,573 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=11220, suffix heap size=132904,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=245 2023-04-12 15:53:44,609 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1681314824595, max timestamp=1681314824602Num uniques 1104; 2023-04-12 15:53:44,610 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=7050, suffix heap size=142784,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=154 2023-04-12 15:53:44,610 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1681314824595, max timestamp=1681314824604Num uniques 1104; 2023-04-12 15:53:44,610 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=7050, suffix heap size=142784,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=154 2023-04-12 15:53:44,612 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681314824595, max timestamp=1681314824604Num uniques 1104; 2023-04-12 15:53:44,612 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=6870, suffix heap size=142784,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=150 2023-04-12 15:53:44,636 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7455, totalHeapSize=17192, min timestamp=1681314824624, max timestamp=1681314824631Num uniques 1013; 2023-04-12 15:53:44,636 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:44,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58414 deadline: 1681314884632, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:44,638 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681314824624, max timestamp=1681314824631Num uniques 1009; 2023-04-12 15:53:44,638 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46208, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-04-12 15:53:44,640 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681314824624, max timestamp=1681314824631Num uniques 1013; 2023-04-12 15:53:44,640 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=6870, suffix heap size=133320,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=150 2023-04-12 15:53:44,636 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46385, new segment data size=7455, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-04-12 15:53:44,647 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=8 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/B/588c2dd62af14c6db4741a912bd136b4 2023-04-12 15:53:44,675 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1681314824650, max timestamp=1681314824669Num uniques 1009; 2023-04-12 15:53:44,675 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1681314824650, max timestamp=1681314824669Num uniques 1022; 2023-04-12 15:53:44,675 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1681314824650, max timestamp=1681314824669Num uniques 1009; 2023-04-12 15:53:44,675 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=10162, suffix heap size=134256,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=222 2023-04-12 15:53:44,675 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=10162, suffix heap size=132904,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=222 2023-04-12 15:53:44,675 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=10162, suffix heap size=132904,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=222 2023-04-12 15:53:44,730 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:44,730 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=781, cellSize=35766, totalHeapSize=81464, min timestamp=1681314824677, max timestamp=1681314824726Num uniques 1081; 2023-04-12 15:53:44,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58340 deadline: 1681314884727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:44,730 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=35766, suffix heap size=140392,new segment heap size=81464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=781 2023-04-12 15:53:44,732 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=781, cellSize=35766, totalHeapSize=81464, min timestamp=1681314824677, max timestamp=1681314824726Num uniques 1081; 2023-04-12 15:53:44,732 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=35766, suffix heap size=140392,new segment heap size=81464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=781 2023-04-12 15:53:44,734 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=781, cellSize=35766, totalHeapSize=81464, min timestamp=1681314824677, max timestamp=1681314824726Num uniques 1081; 2023-04-12 15:53:44,734 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=35766, suffix heap size=140392,new segment heap size=81464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=781 2023-04-12 15:53:44,750 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:44,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58374 deadline: 1681314884744, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:44,751 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:44,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58398 deadline: 1681314884745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:44,752 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:44,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58354 deadline: 1681314884748, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:44,755 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:44,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58414 deadline: 1681314884751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:44,834 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:44,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58340 deadline: 1681314884834, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:44,855 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:44,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58398 deadline: 1681314884854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:44,857 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:44,857 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:44,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58354 deadline: 1681314884856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:44,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58374 deadline: 1681314884854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:44,859 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:44,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58414 deadline: 1681314884859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:45,038 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:45,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58340 deadline: 1681314885038, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:45,060 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:45,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58374 deadline: 1681314885060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:45,062 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:45,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58398 deadline: 1681314885060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:45,063 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:45,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58414 deadline: 1681314885062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:45,068 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:45,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58354 deadline: 1681314885061, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:45,118 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=8 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/C/c5a0d6094fee4311bf254280a3e4b08f 2023-04-12 15:53:45,135 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/A/e029725a1c65412ebf79fbcaab14f2c7 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/e029725a1c65412ebf79fbcaab14f2c7 2023-04-12 15:53:45,151 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/e029725a1c65412ebf79fbcaab14f2c7, entries=100, sequenceid=8, filesize=9.3 K 2023-04-12 15:53:45,159 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/B/588c2dd62af14c6db4741a912bd136b4 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/588c2dd62af14c6db4741a912bd136b4 2023-04-12 15:53:45,202 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/588c2dd62af14c6db4741a912bd136b4, entries=100, sequenceid=8, filesize=9.3 K 2023-04-12 15:53:45,218 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/C/c5a0d6094fee4311bf254280a3e4b08f as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/c5a0d6094fee4311bf254280a3e4b08f 2023-04-12 15:53:45,246 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/c5a0d6094fee4311bf254280a3e4b08f, entries=100, sequenceid=8, filesize=9.3 K 2023-04-12 15:53:45,249 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=174.43 KB/178620 for feda8274e3aaad09f0f3a1336e663d20 in 1320ms, sequenceid=8, compaction requested=false 2023-04-12 15:53:45,251 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-12 15:53:45,256 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for feda8274e3aaad09f0f3a1336e663d20: 2023-04-12 15:53:45,257 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:45,260 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:53:45,261 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:53:45,261 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:45,261 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:53:45,261 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:53:45,277 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:45,278 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:45,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:45,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:45,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:45,281 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:45,281 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:53:45,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:45,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:45,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:45,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:45,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:45,284 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:53:45,306 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a7e9652[Count = 0] remaining members to acquire global barrier 2023-04-12 15:53:45,307 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:53:45,307 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:45,316 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:45,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:45,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:45,317 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:53:45,317 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:53:45,317 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:53:45,317 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:45,317 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:53:45,322 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:53:45,322 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:45,323 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:53:45,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:45,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:45,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:45,323 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:53:45,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:45,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:45,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:45,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:45,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:45,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:45,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:45,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:53:45,329 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:53:45,329 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:53:45,329 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:53:45,329 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:53:45,329 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:53:45,329 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:53:45,332 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:45,332 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:45,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:45,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:45,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:45,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:45,332 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:53:45,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:45,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:45,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:53:45,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:53:45,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:45,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:45,334 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:45,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:45,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:45,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:45,337 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:45,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:45,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:45,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:45,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on feda8274e3aaad09f0f3a1336e663d20 2023-04-12 15:53:45,351 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing feda8274e3aaad09f0f3a1336e663d20 3/3 column families, dataSize=181.14 KB heapSize=466.13 KB 2023-04-12 15:53:45,351 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK feda8274e3aaad09f0f3a1336e663d20, store=A 2023-04-12 15:53:45,355 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:53:45,356 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK feda8274e3aaad09f0f3a1336e663d20, store=B 2023-04-12 15:53:45,356 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:53:45,356 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK feda8274e3aaad09f0f3a1336e663d20, store=C 2023-04-12 15:53:45,356 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:53:45,370 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:53:45,371 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:45,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:53:45,371 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:45,371 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:53:45,371 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:45,371 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:45,371 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:53:45,371 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:53:45,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:53:45,371 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:53:45,371 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:45,372 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:45,372 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:45,373 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:53:45,374 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:53:45,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:53:45,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:53:45,387 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:45,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58354 deadline: 1681314885377, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:45,400 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:45,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58340 deadline: 1681314885384, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:45,401 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:45,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58374 deadline: 1681314885385, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:45,402 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:45,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58398 deadline: 1681314885387, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:45,403 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:45,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58414 deadline: 1681314885389, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:45,461 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60.38 KB at sequenceid=177 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/A/32528e7e7acb46b783961424e951923d 2023-04-12 15:53:45,474 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:53:45,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:53:45,488 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:53:45,504 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:45,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58340 deadline: 1681314885502, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:45,505 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:45,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58398 deadline: 1681314885504, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:45,507 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:45,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58414 deadline: 1681314885506, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:45,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:53:45,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:53:45,508 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:53:45,508 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:53:45,509 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:45,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58374 deadline: 1681314885509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:45,514 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:53:45,514 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:53:45,515 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:45,515 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:45,516 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:53:45,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:53:45,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:53:45,517 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:45,524 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:45,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:45,524 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:53:45,525 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:45,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:53:45,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:45,526 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:53:45,529 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:53:45,529 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:53:45,529 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:53:45,529 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:53:45,540 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:53:45,541 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:53:45,541 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:53:45,541 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:45,541 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. started... 2023-04-12 15:53:45,541 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. as already flushing 2023-04-12 15:53:45,541 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:45,548 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:53:45,549 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:53:45,549 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:45,550 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:53:45,550 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:53:45,556 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:45,556 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:45,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:45,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:45,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:45,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:45,558 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60.38 KB at sequenceid=177 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/B/f29c6b6646e74a54934655a82b02b1e9 2023-04-12 15:53:45,564 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:45,564 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:53:45,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:45,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:45,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:45,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:45,566 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:53:45,567 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5faddf99[Count = 0] remaining members to acquire global barrier 2023-04-12 15:53:45,567 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:53:45,567 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:45,569 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:45,569 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:53:45,569 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:45,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:45,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:45,569 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:53:45,569 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:53:45,569 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:53:45,584 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:53:45,584 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:45,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:45,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:45,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:45,584 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:53:45,585 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:53:45,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:45,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:45,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:45,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:45,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:45,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:45,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:45,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:53:45,589 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:53:45,590 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:53:45,590 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:53:45,590 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:53:45,590 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:53:45,590 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:53:45,599 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:45,599 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:53:45,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:45,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:45,599 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:45,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:45,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:45,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:45,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:53:45,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:53:45,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:45,600 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:45,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:45,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:45,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:45,602 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:45,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:45,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:45,627 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:45,628 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:45,628 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:53:45,628 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:53:45,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:53:45,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:53:45,628 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:53:45,628 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:45,628 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:53:45,628 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:53:45,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:45,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:53:45,629 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:53:45,628 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:45,629 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:45,629 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:45,629 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:45,629 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:53:45,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:53:45,630 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:53:45,657 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60.38 KB at sequenceid=177 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/C/96022ac6fe824abebf8ed0dd0de3cd7c 2023-04-12 15:53:45,670 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/A/32528e7e7acb46b783961424e951923d as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/32528e7e7acb46b783961424e951923d 2023-04-12 15:53:45,683 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/32528e7e7acb46b783961424e951923d, entries=200, sequenceid=177, filesize=14.1 K 2023-04-12 15:53:45,686 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/B/f29c6b6646e74a54934655a82b02b1e9 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/f29c6b6646e74a54934655a82b02b1e9 2023-04-12 15:53:45,710 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:45,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58340 deadline: 1681314885707, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:45,711 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/f29c6b6646e74a54934655a82b02b1e9, entries=150, sequenceid=177, filesize=11.8 K 2023-04-12 15:53:45,711 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:45,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58398 deadline: 1681314885710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:45,714 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:45,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58414 deadline: 1681314885711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:45,715 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/C/96022ac6fe824abebf8ed0dd0de3cd7c as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/96022ac6fe824abebf8ed0dd0de3cd7c 2023-04-12 15:53:45,721 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:45,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58374 deadline: 1681314885713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:45,730 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:53:45,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:53:45,732 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/96022ac6fe824abebf8ed0dd0de3cd7c, entries=150, sequenceid=177, filesize=11.8 K 2023-04-12 15:53:45,742 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~181.14 KB/185490, heapSize ~466.08 KB/477264, currentSize=26.84 KB/27480 for feda8274e3aaad09f0f3a1336e663d20 in 391ms, sequenceid=177, compaction requested=false 2023-04-12 15:53:45,742 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:53:45,743 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for feda8274e3aaad09f0f3a1336e663d20: 2023-04-12 15:53:45,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:53:45,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:53:45,752 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:53:45,752 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:53:45,756 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:53:45,756 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:53:45,757 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:45,757 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:45,759 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:53:45,759 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:45,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:53:45,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:53:45,760 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:45,760 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:53:45,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:45,761 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:45,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:53:45,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:45,762 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:53:45,762 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:53:45,762 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:53:45,763 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:53:45,763 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:53:45,764 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:53:45,766 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:53:45,766 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:53:45,766 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:45,766 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. started... 2023-04-12 15:53:45,766 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing feda8274e3aaad09f0f3a1336e663d20 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-12 15:53:45,767 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK feda8274e3aaad09f0f3a1336e663d20, store=A 2023-04-12 15:53:45,767 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:53:45,767 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK feda8274e3aaad09f0f3a1336e663d20, store=B 2023-04-12 15:53:45,767 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:53:45,767 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK feda8274e3aaad09f0f3a1336e663d20, store=C 2023-04-12 15:53:45,767 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:53:45,809 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=184 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/A/025d1df469b14594a716bcec44ee239b 2023-04-12 15:53:45,856 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=184 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/B/29368c91015e481ea5a07516ea99be74 2023-04-12 15:53:45,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on feda8274e3aaad09f0f3a1336e663d20 2023-04-12 15:53:45,917 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. as already flushing 2023-04-12 15:53:45,983 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-04-12 15:53:45,984 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-04-12 15:53:46,001 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681314825940, max timestamp=1681314825998Num uniques 859; 2023-04-12 15:53:46,001 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-12 15:53:46,001 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681314825940, max timestamp=1681314825998Num uniques 859; 2023-04-12 15:53:46,001 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-12 15:53:46,003 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681314825973, max timestamp=1681314825998Num uniques 859; 2023-04-12 15:53:46,003 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-12 15:53:46,045 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681314826026, max timestamp=1681314826042Num uniques 1068; 2023-04-12 15:53:46,045 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-04-12 15:53:46,045 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681314826026, max timestamp=1681314826042Num uniques 1068; 2023-04-12 15:53:46,045 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-04-12 15:53:46,047 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681314826026, max timestamp=1681314826042Num uniques 1018; 2023-04-12 15:53:46,047 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-12 15:53:46,134 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681314826073, max timestamp=1681314826130Num uniques 1127; 2023-04-12 15:53:46,134 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681314826073, max timestamp=1681314826130Num uniques 1027; 2023-04-12 15:53:46,134 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-12 15:53:46,134 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=8102, suffix heap size=145176,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=177 2023-04-12 15:53:46,134 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681314826073, max timestamp=1681314826130Num uniques 1127; 2023-04-12 15:53:46,135 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=8102, suffix heap size=145176,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=177 2023-04-12 15:53:46,172 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=336, cellSize=15386, totalHeapSize=35184, min timestamp=1681314826148, max timestamp=1681314826168Num uniques 1036; 2023-04-12 15:53:46,172 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=336, cellSize=15386, totalHeapSize=35184, min timestamp=1681314826148, max timestamp=1681314826169Num uniques 1036; 2023-04-12 15:53:46,172 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=15386, suffix heap size=135712,new segment heap size=35184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=336 2023-04-12 15:53:46,173 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=15386, suffix heap size=135712,new segment heap size=35184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=336 2023-04-12 15:53:46,176 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681314826151, max timestamp=1681314826169Num uniques 1036; 2023-04-12 15:53:46,176 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47437, new segment data size=6870, suffix heap size=135712,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=150 2023-04-12 15:53:46,209 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681314826192, max timestamp=1681314826205Num uniques 1195; 2023-04-12 15:53:46,210 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=54730, new segment data size=8930, suffix heap size=152248,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1195, new segment cells count=195 2023-04-12 15:53:46,212 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681314826192, max timestamp=1681314826205Num uniques 1195; 2023-04-12 15:53:46,212 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=54730, new segment data size=8930, suffix heap size=152248,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1195, new segment cells count=195 2023-04-12 15:53:46,213 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681314826192, max timestamp=1681314826205Num uniques 1009; 2023-04-12 15:53:46,213 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46223, new segment data size=8930, suffix heap size=132904,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=195 2023-04-12 15:53:46,310 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=184 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/C/648b4f8cb5934d67aa4c31ad2977e774 2023-04-12 15:53:46,321 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1681314826288, max timestamp=1681314826317Num uniques 1054; 2023-04-12 15:53:46,321 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1681314826288, max timestamp=1681314826317Num uniques 1054; 2023-04-12 15:53:46,321 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-12 15:53:46,321 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-12 15:53:46,324 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1681314826288, max timestamp=1681314826317Num uniques 1054; 2023-04-12 15:53:46,324 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-12 15:53:46,328 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/A/025d1df469b14594a716bcec44ee239b as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/025d1df469b14594a716bcec44ee239b 2023-04-12 15:53:46,349 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/025d1df469b14594a716bcec44ee239b, entries=150, sequenceid=184, filesize=11.8 K 2023-04-12 15:53:46,353 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/B/29368c91015e481ea5a07516ea99be74 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/29368c91015e481ea5a07516ea99be74 2023-04-12 15:53:46,368 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1681314826337, max timestamp=1681314826365Num uniques 1013; 2023-04-12 15:53:46,370 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=9748, suffix heap size=133320,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=213 2023-04-12 15:53:46,370 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/29368c91015e481ea5a07516ea99be74, entries=150, sequenceid=184, filesize=11.8 K 2023-04-12 15:53:46,372 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1681314826337, max timestamp=1681314826365Num uniques 1013; 2023-04-12 15:53:46,372 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=9748, suffix heap size=133320,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=213 2023-04-12 15:53:46,374 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/C/648b4f8cb5934d67aa4c31ad2977e774 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/648b4f8cb5934d67aa4c31ad2977e774 2023-04-12 15:53:46,375 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1681314826357, max timestamp=1681314826365Num uniques 1013; 2023-04-12 15:53:46,375 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-04-12 15:53:46,394 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/648b4f8cb5934d67aa4c31ad2977e774, entries=150, sequenceid=184, filesize=11.8 K 2023-04-12 15:53:46,436 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=78.27 KB/80150 for feda8274e3aaad09f0f3a1336e663d20 in 670ms, sequenceid=184, compaction requested=true 2023-04-12 15:53:46,437 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for feda8274e3aaad09f0f3a1336e663d20: 2023-04-12 15:53:46,437 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:46,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on feda8274e3aaad09f0f3a1336e663d20 2023-04-12 15:53:46,437 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:53:46,437 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:53:46,437 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:46,437 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:53:46,437 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:53:46,437 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing feda8274e3aaad09f0f3a1336e663d20 3/3 column families, dataSize=98.40 KB heapSize=275.23 KB 2023-04-12 15:53:46,441 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK feda8274e3aaad09f0f3a1336e663d20, store=A 2023-04-12 15:53:46,441 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:53:46,441 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK feda8274e3aaad09f0f3a1336e663d20, store=B 2023-04-12 15:53:46,441 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:53:46,441 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK feda8274e3aaad09f0f3a1336e663d20, store=C 2023-04-12 15:53:46,441 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:53:46,442 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:46,442 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:46,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:46,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:46,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:46,443 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:46,443 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:53:46,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:46,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:46,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:46,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:46,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:46,445 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:53:46,445 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1760c9be[Count = 0] remaining members to acquire global barrier 2023-04-12 15:53:46,445 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:53:46,445 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:46,448 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:46,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:46,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:46,448 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:53:46,448 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:53:46,448 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:53:46,453 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:53:46,453 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:46,453 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:53:46,453 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:53:46,454 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:53:46,454 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:53:46,454 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:53:46,454 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:53:46,454 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:53:46,454 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:53:46,454 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:53:46,458 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:46,458 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:53:46,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:46,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:46,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:53:46,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:53:46,461 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:46,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:46,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:46,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:46,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:46,462 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:46,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:46,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:46,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:46,473 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:46,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:46,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:46,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:46,493 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:53:46,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:46,493 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:46,493 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:53:46,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:53:46,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:53:46,494 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:46,499 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:46,499 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:46,499 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:46,499 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:46,499 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:46,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:53:46,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:53:46,501 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:53:46,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:53:46,499 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:53:46,503 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:53:46,503 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:53:46,503 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:53:46,523 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:46,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58354 deadline: 1681314886507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:46,524 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:46,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58414 deadline: 1681314886516, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:46,533 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:46,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58374 deadline: 1681314886523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:46,534 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:46,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58340 deadline: 1681314886523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:46,543 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:46,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58398 deadline: 1681314886525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:46,603 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:53:46,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:53:46,611 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:53:46,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:53:46,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:53:46,615 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:53:46,615 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:53:46,621 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:53:46,621 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:53:46,622 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:46,622 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:46,623 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:53:46,623 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:46,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:53:46,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:53:46,624 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:46,624 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:53:46,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:46,624 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:46,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:53:46,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:46,625 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:53:46,626 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:53:46,626 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:53:46,626 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:53:46,626 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:53:46,627 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:53:46,628 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:53:46,628 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:53:46,628 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:46,628 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. started... 2023-04-12 15:53:46,628 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. as already flushing 2023-04-12 15:53:46,628 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:46,633 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:53:46,633 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:53:46,633 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:46,633 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:53:46,633 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:53:46,635 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:46,635 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:46,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:46,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:46,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:46,637 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:46,640 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:53:46,641 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:46,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58414 deadline: 1681314886628, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:46,659 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:46,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58340 deadline: 1681314886644, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:46,660 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:46,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58398 deadline: 1681314886646, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:46,654 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:46,647 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:46,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:46,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58354 deadline: 1681314886628, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:46,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:46,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:46,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:46,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:46,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58374 deadline: 1681314886636, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:46,664 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:53:46,664 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ed46a78[Count = 0] remaining members to acquire global barrier 2023-04-12 15:53:46,664 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:53:46,664 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:46,666 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:46,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:46,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:46,666 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:53:46,666 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:53:46,666 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:53:46,666 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:46,666 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:53:46,673 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:53:46,673 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:53:46,673 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:46,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:46,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:46,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:46,673 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:53:46,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:46,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:46,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:46,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:46,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:46,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:46,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:46,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:53:46,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:53:46,694 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:53:46,694 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:53:46,694 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:53:46,694 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:53:46,694 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:53:46,696 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:46,696 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:53:46,696 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:46,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:46,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:46,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:53:46,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:53:46,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:46,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:46,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:46,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:46,699 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:46,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:46,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:46,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:46,703 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:46,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:46,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:46,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:46,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:46,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:46,717 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:53:46,718 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:53:46,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:53:46,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:53:46,718 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:46,718 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:46,718 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:53:46,718 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:53:46,719 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:53:46,718 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:46,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:53:46,719 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:46,720 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:46,720 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:46,720 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:46,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:53:46,720 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:53:46,720 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:53:46,721 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:53:46,786 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-12 15:53:46,820 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:53:46,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:53:46,827 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:53:46,875 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:46,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58340 deadline: 1681314886867, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:46,876 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:46,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58414 deadline: 1681314886874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:46,877 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:46,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58354 deadline: 1681314886874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:46,878 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:46,878 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:46,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58374 deadline: 1681314886875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:46,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58398 deadline: 1681314886875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:46,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:53:46,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:53:46,892 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:53:46,892 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:53:46,894 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:53:46,894 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:53:46,894 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:46,894 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:46,896 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:46,896 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:53:46,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:53:46,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:53:46,897 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:46,897 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:53:46,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:46,901 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:46,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:53:46,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:46,901 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:53:46,906 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:53:46,906 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:53:46,906 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:53:46,906 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:53:46,911 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:53:46,912 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:53:46,912 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:53:46,915 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=320 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/A/e5f68fdf2e5a4936b59f3cfbbfce051d 2023-04-12 15:53:46,917 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:46,917 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. started... 2023-04-12 15:53:46,917 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. as already flushing 2023-04-12 15:53:46,917 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:46,917 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:53:46,917 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:53:46,918 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:46,918 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:53:46,918 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:53:46,922 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:46,922 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:46,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:46,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:46,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:46,922 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:46,922 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:53:46,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:46,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:46,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:46,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:46,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:46,927 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:53:46,927 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3026b1f6[Count = 0] remaining members to acquire global barrier 2023-04-12 15:53:46,927 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:53:46,927 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:46,933 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:46,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:46,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:46,933 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:53:46,933 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:53:46,933 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:53:46,933 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:46,934 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:53:46,935 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:53:46,935 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:53:46,935 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:53:46,936 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:46,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:46,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:46,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:46,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:46,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:46,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:46,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:46,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:46,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:46,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:46,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:53:46,944 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:53:46,945 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:53:46,945 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:53:46,945 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:53:46,945 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:53:46,945 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:53:46,960 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:46,961 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:46,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:46,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:46,961 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:53:46,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:46,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:46,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:46,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:53:46,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:53:46,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:46,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:46,963 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:46,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:46,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:46,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:46,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:46,975 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:46,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:46,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:46,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:46,982 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:46,982 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:53:46,982 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:53:46,982 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:46,982 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:46,982 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:46,982 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:46,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:53:46,982 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:53:46,983 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:53:46,982 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:53:46,983 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:53:46,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:53:46,982 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:46,983 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:46,983 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:53:46,983 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:53:46,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:53:46,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:53:47,083 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:53:47,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:53:47,088 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:53:47,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:53:47,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:53:47,124 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:53:47,124 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:53:47,129 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:53:47,129 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:53:47,129 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:47,130 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:47,131 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:53:47,131 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:47,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:53:47,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:53:47,132 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:47,132 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:53:47,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:47,132 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:47,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:53:47,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:47,133 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:53:47,133 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:53:47,133 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:53:47,133 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:53:47,133 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:53:47,135 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:53:47,135 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:53:47,135 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:53:47,135 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:47,135 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. started... 2023-04-12 15:53:47,135 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. as already flushing 2023-04-12 15:53:47,135 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:47,135 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:53:47,136 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:53:47,136 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:47,136 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:53:47,136 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:53:47,138 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:47,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:47,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:47,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:47,139 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:47,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:47,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:47,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:47,141 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:47,142 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:53:47,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:47,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:47,143 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:53:47,143 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c6e17c[Count = 0] remaining members to acquire global barrier 2023-04-12 15:53:47,143 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:53:47,143 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:47,145 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:47,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:47,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:47,146 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:53:47,146 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:53:47,146 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:53:47,147 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:47,147 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:53:47,148 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:53:47,148 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:53:47,148 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:53:47,151 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:47,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:47,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:47,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:47,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:47,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:47,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:47,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:47,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:47,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:47,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:47,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:53:47,156 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:53:47,156 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:53:47,156 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:53:47,156 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:53:47,156 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:53:47,156 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:53:47,158 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:47,158 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:47,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:47,158 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:53:47,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:47,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:47,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:47,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:47,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:53:47,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:53:47,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:47,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:47,160 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:47,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:47,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:47,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:47,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:47,162 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:47,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:47,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:47,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:47,166 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:53:47,166 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:53:47,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:53:47,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:53:47,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:53:47,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:53:47,168 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:47,168 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:47,168 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:47,168 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:47,168 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:47,168 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:47,168 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:47,168 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:53:47,168 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:53:47,170 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:53:47,170 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:53:47,170 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:53:47,181 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:47,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58340 deadline: 1681314887177, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:47,182 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:47,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58414 deadline: 1681314887180, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:47,183 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:47,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58354 deadline: 1681314887181, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:47,191 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:47,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58398 deadline: 1681314887186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:47,192 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:47,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58374 deadline: 1681314887186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:47,270 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:53:47,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:53:47,274 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:53:47,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:53:47,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:53:47,278 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:53:47,278 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:53:47,281 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:53:47,281 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:53:47,281 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:47,282 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:47,285 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:53:47,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:53:47,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:53:47,286 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:47,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:47,286 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:47,286 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:53:47,287 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:47,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:53:47,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:47,288 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:53:47,288 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:53:47,288 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:53:47,288 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:53:47,288 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:53:47,289 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:53:47,289 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:53:47,289 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:53:47,289 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:47,290 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. started... 2023-04-12 15:53:47,290 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. as already flushing 2023-04-12 15:53:47,290 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:47,290 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:53:47,290 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:53:47,290 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:47,290 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:53:47,290 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:53:47,292 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:47,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:47,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:47,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:47,292 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:47,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:47,293 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:47,293 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:53:47,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:47,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:47,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:47,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:47,296 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:53:47,296 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@107af5c3[Count = 0] remaining members to acquire global barrier 2023-04-12 15:53:47,296 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:53:47,297 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:47,299 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:47,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:47,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:47,299 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:53:47,299 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:53:47,299 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:53:47,300 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:47,300 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:53:47,301 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:53:47,301 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:53:47,301 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:53:47,302 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:47,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:47,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:47,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:47,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:47,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:47,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:47,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:47,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:47,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:47,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:47,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:53:47,308 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:53:47,308 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:53:47,309 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:53:47,309 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:53:47,309 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:53:47,309 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:53:47,311 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:47,311 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:47,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:47,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:47,311 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:53:47,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:47,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:47,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:47,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:53:47,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:53:47,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:47,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:47,313 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:47,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:47,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:47,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:47,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:47,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:47,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:47,317 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:47,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:47,321 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:53:47,321 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:53:47,321 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:53:47,321 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:47,321 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:47,321 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:47,321 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:47,321 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:47,321 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:47,321 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:47,321 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:53:47,325 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:53:47,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:53:47,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:53:47,325 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:53:47,325 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:53:47,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:53:47,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:53:47,411 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=320 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/B/f02085ee38e44d74bf5c59e9f6f4089f 2023-04-12 15:53:47,426 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:53:47,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:53:47,432 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:53:47,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:53:47,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:53:47,452 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:53:47,452 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:53:47,457 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:53:47,457 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:53:47,457 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:47,458 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:47,459 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:53:47,459 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:47,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:53:47,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:53:47,460 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:47,460 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:53:47,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:47,462 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:47,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:53:47,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:47,463 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:53:47,468 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:53:47,468 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:53:47,468 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:53:47,468 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:53:47,469 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:53:47,470 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:53:47,470 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:53:47,470 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:47,470 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. started... 2023-04-12 15:53:47,470 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. as already flushing 2023-04-12 15:53:47,470 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:47,470 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:53:47,470 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:53:47,470 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:47,470 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:53:47,470 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:53:47,472 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:47,472 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:47,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:47,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:47,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:47,473 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:47,473 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:53:47,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:47,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:47,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:47,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:47,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:47,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:53:47,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3dd0ca1e[Count = 0] remaining members to acquire global barrier 2023-04-12 15:53:47,475 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:53:47,475 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:47,478 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:47,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:47,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:47,478 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:53:47,478 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:53:47,478 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:53:47,478 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:47,478 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:53:47,480 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:47,480 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:53:47,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:47,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:47,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:47,480 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:53:47,480 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:53:47,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:47,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:47,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:47,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:47,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:47,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:47,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:47,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:53:47,483 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:53:47,483 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:53:47,483 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:53:47,483 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:53:47,483 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:53:47,483 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:53:47,485 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:47,486 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:53:47,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:47,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:47,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:53:47,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:53:47,486 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:47,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:47,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:47,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:47,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:47,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:47,504 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:47,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:47,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:47,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:47,506 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:47,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:47,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:47,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:47,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:47,513 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:47,514 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:47,514 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:47,514 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:53:47,514 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:53:47,514 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:47,514 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:53:47,514 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:53:47,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:53:47,514 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:53:47,514 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:53:47,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:53:47,514 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:53:47,515 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:53:47,514 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:47,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:53:47,515 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:47,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:53:47,515 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:47,526 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=320 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/C/1e1c7c7b609441d083d9b948e0170c06 2023-04-12 15:53:47,536 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/A/e5f68fdf2e5a4936b59f3cfbbfce051d as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/e5f68fdf2e5a4936b59f3cfbbfce051d 2023-04-12 15:53:47,546 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/e5f68fdf2e5a4936b59f3cfbbfce051d, entries=250, sequenceid=320, filesize=16.7 K 2023-04-12 15:53:47,548 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/B/f02085ee38e44d74bf5c59e9f6f4089f as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/f02085ee38e44d74bf5c59e9f6f4089f 2023-04-12 15:53:47,587 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/f02085ee38e44d74bf5c59e9f6f4089f, entries=150, sequenceid=320, filesize=11.9 K 2023-04-12 15:53:47,598 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/C/1e1c7c7b609441d083d9b948e0170c06 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/1e1c7c7b609441d083d9b948e0170c06 2023-04-12 15:53:47,630 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:53:47,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:53:47,636 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:53:47,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:53:47,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:53:47,638 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:53:47,638 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:53:47,641 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:53:47,641 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:53:47,641 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:47,641 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:47,643 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/1e1c7c7b609441d083d9b948e0170c06, entries=150, sequenceid=320, filesize=11.9 K 2023-04-12 15:53:47,645 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:53:47,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:53:47,645 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:47,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:53:47,646 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:47,646 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:53:47,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:47,646 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~111.82 KB/114500, heapSize ~315.03 KB/322592, currentSize=73.80 KB/75570 for feda8274e3aaad09f0f3a1336e663d20 in 1209ms, sequenceid=320, compaction requested=true 2023-04-12 15:53:47,646 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for feda8274e3aaad09f0f3a1336e663d20: 2023-04-12 15:53:47,646 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:47,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:53:47,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:47,647 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:53:47,662 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:53:47,663 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:53:47,663 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:53:47,663 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:53:47,674 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:53:47,674 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:53:47,674 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-12 15:53:47,675 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:53:47,675 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:53:47,676 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:53:47,676 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-12 15:53:47,676 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:53:47,677 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:47,677 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. started... 2023-04-12 15:53:47,677 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing feda8274e3aaad09f0f3a1336e663d20 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-12 15:53:47,677 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK feda8274e3aaad09f0f3a1336e663d20, store=A 2023-04-12 15:53:47,677 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:53:47,677 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK feda8274e3aaad09f0f3a1336e663d20, store=B 2023-04-12 15:53:47,678 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:53:47,678 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK feda8274e3aaad09f0f3a1336e663d20, store=C 2023-04-12 15:53:47,679 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:53:47,685 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45800 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-12 15:53:47,687 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1912): feda8274e3aaad09f0f3a1336e663d20/B is initiating minor compaction (all files) 2023-04-12 15:53:47,687 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2259): Starting compaction of feda8274e3aaad09f0f3a1336e663d20/B in TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:47,687 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/588c2dd62af14c6db4741a912bd136b4, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/f29c6b6646e74a54934655a82b02b1e9, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/29368c91015e481ea5a07516ea99be74, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/f02085ee38e44d74bf5c59e9f6f4089f] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp, totalSize=44.7 K 2023-04-12 15:53:47,691 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 588c2dd62af14c6db4741a912bd136b4, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=8, earliestPutTs=1681314823878 2023-04-12 15:53:47,692 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-12 15:53:47,694 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. as already flushing 2023-04-12 15:53:47,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on feda8274e3aaad09f0f3a1336e663d20 2023-04-12 15:53:47,695 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting f29c6b6646e74a54934655a82b02b1e9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=177, earliestPutTs=1681314824741 2023-04-12 15:53:47,696 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 53070 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-12 15:53:47,696 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1912): feda8274e3aaad09f0f3a1336e663d20/A is initiating minor compaction (all files) 2023-04-12 15:53:47,696 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of feda8274e3aaad09f0f3a1336e663d20/A in TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:47,697 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/e029725a1c65412ebf79fbcaab14f2c7, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/32528e7e7acb46b783961424e951923d, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/025d1df469b14594a716bcec44ee239b, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/e5f68fdf2e5a4936b59f3cfbbfce051d] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp, totalSize=51.8 K 2023-04-12 15:53:47,697 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-04-12 15:53:47,699 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting e029725a1c65412ebf79fbcaab14f2c7, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=8, earliestPutTs=1681314823878 2023-04-12 15:53:47,704 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 29368c91015e481ea5a07516ea99be74, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=184, earliestPutTs=1681314825380 2023-04-12 15:53:47,704 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 32528e7e7acb46b783961424e951923d, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=177, earliestPutTs=1681314824741 2023-04-12 15:53:47,705 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting f02085ee38e44d74bf5c59e9f6f4089f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=320, earliestPutTs=1681314826375 2023-04-12 15:53:47,706 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 025d1df469b14594a716bcec44ee239b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=184, earliestPutTs=1681314825380 2023-04-12 15:53:47,708 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting e5f68fdf2e5a4936b59f3cfbbfce051d, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=320, earliestPutTs=1681314826375 2023-04-12 15:53:47,748 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=334 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/A/6c9aeb7cb4bc41d3b501757541eb2baf 2023-04-12 15:53:47,769 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] throttle.PressureAwareThroughputController(145): feda8274e3aaad09f0f3a1336e663d20#A#compaction#13 average throughput is 0.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:53:47,774 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] throttle.PressureAwareThroughputController(145): feda8274e3aaad09f0f3a1336e663d20#B#compaction#14 average throughput is 1.09 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:53:47,812 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:47,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58354 deadline: 1681314887749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:47,813 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:47,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58340 deadline: 1681314887754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:47,840 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:47,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58414 deadline: 1681314887812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:47,841 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:47,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58374 deadline: 1681314887818, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:47,856 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:47,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58398 deadline: 1681314887840, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:47,866 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=334 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/B/73fd13d4bc004b969365cce1d296fbb4 2023-04-12 15:53:47,882 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/B/59cfd2bedaaa4e0688bb8f241c4d75d3 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/59cfd2bedaaa4e0688bb8f241c4d75d3 2023-04-12 15:53:47,921 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in feda8274e3aaad09f0f3a1336e663d20/B of feda8274e3aaad09f0f3a1336e663d20 into 59cfd2bedaaa4e0688bb8f241c4d75d3(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:53:47,929 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for feda8274e3aaad09f0f3a1336e663d20: 2023-04-12 15:53:47,929 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20., storeName=feda8274e3aaad09f0f3a1336e663d20/B, priority=12, startTime=1681314827674; duration=0sec 2023-04-12 15:53:47,934 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:53:47,935 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-12 15:53:47,937 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:47,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58354 deadline: 1681314887914, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:47,937 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45800 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-12 15:53:47,937 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1912): feda8274e3aaad09f0f3a1336e663d20/C is initiating minor compaction (all files) 2023-04-12 15:53:47,937 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:47,937 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2259): Starting compaction of feda8274e3aaad09f0f3a1336e663d20/C in TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:47,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58340 deadline: 1681314887915, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:47,938 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/c5a0d6094fee4311bf254280a3e4b08f, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/96022ac6fe824abebf8ed0dd0de3cd7c, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/648b4f8cb5934d67aa4c31ad2977e774, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/1e1c7c7b609441d083d9b948e0170c06] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp, totalSize=44.7 K 2023-04-12 15:53:47,939 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting c5a0d6094fee4311bf254280a3e4b08f, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=8, earliestPutTs=1681314823878 2023-04-12 15:53:47,939 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 96022ac6fe824abebf8ed0dd0de3cd7c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=177, earliestPutTs=1681314824741 2023-04-12 15:53:47,940 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 648b4f8cb5934d67aa4c31ad2977e774, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=184, earliestPutTs=1681314825380 2023-04-12 15:53:47,941 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 1e1c7c7b609441d083d9b948e0170c06, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=320, earliestPutTs=1681314826375 2023-04-12 15:53:47,943 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=334 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/C/bb83c8786ab54e3fb162fac4fe55e697 2023-04-12 15:53:47,944 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:47,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58374 deadline: 1681314887943, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:47,948 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:47,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58414 deadline: 1681314887947, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:47,955 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/A/6c9aeb7cb4bc41d3b501757541eb2baf as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/6c9aeb7cb4bc41d3b501757541eb2baf 2023-04-12 15:53:47,970 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:47,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58398 deadline: 1681314887966, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:47,988 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/6c9aeb7cb4bc41d3b501757541eb2baf, entries=150, sequenceid=334, filesize=11.9 K 2023-04-12 15:53:47,996 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] throttle.PressureAwareThroughputController(145): feda8274e3aaad09f0f3a1336e663d20#C#compaction#17 average throughput is 0.50 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:53:48,003 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/B/73fd13d4bc004b969365cce1d296fbb4 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/73fd13d4bc004b969365cce1d296fbb4 2023-04-12 15:53:48,033 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/73fd13d4bc004b969365cce1d296fbb4, entries=150, sequenceid=334, filesize=11.9 K 2023-04-12 15:53:48,034 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/C/bb83c8786ab54e3fb162fac4fe55e697 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/bb83c8786ab54e3fb162fac4fe55e697 2023-04-12 15:53:48,047 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/bb83c8786ab54e3fb162fac4fe55e697, entries=150, sequenceid=334, filesize=11.9 K 2023-04-12 15:53:48,049 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=114.05 KB/116790 for feda8274e3aaad09f0f3a1336e663d20 in 372ms, sequenceid=334, compaction requested=false 2023-04-12 15:53:48,049 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for feda8274e3aaad09f0f3a1336e663d20: 2023-04-12 15:53:48,049 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:48,049 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:53:48,050 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:53:48,050 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,050 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:53:48,050 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:53:48,053 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:48,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:48,053 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:48,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:48,054 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:48,054 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:53:48,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:48,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:48,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:48,056 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:53:48,056 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38fc7553[Count = 0] remaining members to acquire global barrier 2023-04-12 15:53:48,056 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:53:48,056 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:48,058 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:48,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:48,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:48,058 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:53:48,058 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:53:48,058 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,058 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:53:48,058 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:53:48,061 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:53:48,061 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:53:48,061 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:53:48,061 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:48,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:48,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:48,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:48,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:48,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:48,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:48,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:53:48,067 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:53:48,067 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:53:48,067 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:53:48,067 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:53:48,067 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:53:48,067 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:53:48,071 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:48,071 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:53:48,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:48,071 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:48,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:48,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:53:48,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:53:48,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:48,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:48,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:48,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:48,073 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:48,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:48,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:48,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:48,075 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:48,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:48,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,080 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:53:48,080 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:53:48,080 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,080 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:48,080 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:48,080 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:53:48,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:53:48,081 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:53:48,080 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:48,081 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:48,081 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:48,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:53:48,081 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:53:48,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:53:48,080 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:53:48,081 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:53:48,084 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:53:48,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:53:48,143 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing feda8274e3aaad09f0f3a1336e663d20 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-12 15:53:48,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on feda8274e3aaad09f0f3a1336e663d20 2023-04-12 15:53:48,144 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681314827762, max timestamp=1681314828141Num uniques 859; 2023-04-12 15:53:48,144 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681314827762, max timestamp=1681314828141Num uniques 859; 2023-04-12 15:53:48,144 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681314827762, max timestamp=1681314828141Num uniques 859; 2023-04-12 15:53:48,144 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-12 15:53:48,144 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-12 15:53:48,144 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-12 15:53:48,144 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK feda8274e3aaad09f0f3a1336e663d20, store=A 2023-04-12 15:53:48,144 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:53:48,145 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK feda8274e3aaad09f0f3a1336e663d20, store=B 2023-04-12 15:53:48,145 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:53:48,145 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK feda8274e3aaad09f0f3a1336e663d20, store=C 2023-04-12 15:53:48,145 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:53:48,184 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:53:48,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:53:48,188 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:53:48,189 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1681314828173, max timestamp=1681314828186Num uniques 859; 2023-04-12 15:53:48,189 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-12 15:53:48,190 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1681314828173, max timestamp=1681314828186Num uniques 859; 2023-04-12 15:53:48,190 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-12 15:53:48,190 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1681314828173, max timestamp=1681314828186Num uniques 859; 2023-04-12 15:53:48,190 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-12 15:53:48,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:53:48,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:53:48,196 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:53:48,196 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:53:48,201 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:53:48,201 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:53:48,201 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:48,202 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:48,203 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:53:48,203 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:53:48,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:53:48,204 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,204 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:53:48,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:48,204 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:48,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:53:48,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:48,205 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:53:48,206 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:53:48,206 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:53:48,206 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:53:48,206 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:53:48,209 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:53:48,211 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:53:48,211 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:53:48,214 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:48,214 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. started... 2023-04-12 15:53:48,214 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. as already flushing 2023-04-12 15:53:48,214 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:48,214 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:53:48,214 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:53:48,214 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,214 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:53:48,214 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:53:48,217 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,217 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:48,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:48,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:48,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:48,218 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:48,218 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:53:48,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:48,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:48,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:48,221 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:53:48,221 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e07be31[Count = 0] remaining members to acquire global barrier 2023-04-12 15:53:48,221 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:53:48,221 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:48,222 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:48,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:48,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:48,223 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:53:48,223 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,223 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:53:48,223 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:53:48,223 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:53:48,224 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,224 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:53:48,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:48,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:48,224 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:53:48,225 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:53:48,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:48,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:48,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:48,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:48,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:48,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:53:48,229 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:53:48,229 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:53:48,230 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:53:48,230 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:53:48,230 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:53:48,230 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:53:48,232 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:48,232 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:53:48,232 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:48,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:48,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:48,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:48,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:48,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:48,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:48,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:53:48,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:53:48,233 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:48,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:48,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:48,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:48,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,234 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:48,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:48,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,238 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,238 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:48,238 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:48,238 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:53:48,238 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:53:48,238 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:53:48,238 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:53:48,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:53:48,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:53:48,238 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:48,238 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:53:48,239 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:48,239 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:48,239 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:53:48,239 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:53:48,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:53:48,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:53:48,250 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/A/dce1631ad0724f95b7b4c1386ccfdaa9 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/dce1631ad0724f95b7b4c1386ccfdaa9 2023-04-12 15:53:48,265 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in feda8274e3aaad09f0f3a1336e663d20/A of feda8274e3aaad09f0f3a1336e663d20 into dce1631ad0724f95b7b4c1386ccfdaa9(size=12.0 K), total size for store is 23.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:53:48,266 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for feda8274e3aaad09f0f3a1336e663d20: 2023-04-12 15:53:48,266 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20., storeName=feda8274e3aaad09f0f3a1336e663d20/A, priority=12, startTime=1681314827648; duration=0sec 2023-04-12 15:53:48,266 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:53:48,291 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:48,291 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:48,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58374 deadline: 1681314888283, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58398 deadline: 1681314888284, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,292 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:48,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58354 deadline: 1681314888286, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,294 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:48,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58414 deadline: 1681314888289, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,294 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:48,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58340 deadline: 1681314888289, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,339 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:53:48,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:53:48,343 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:53:48,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:53:48,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:53:48,345 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:53:48,346 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:53:48,354 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:53:48,354 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:53:48,356 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:48,356 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:48,358 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:53:48,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:53:48,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:53:48,358 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:48,360 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:48,361 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:53:48,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:48,361 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:53:48,361 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:53:48,361 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:53:48,361 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:53:48,362 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:53:48,362 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:53:48,363 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:53:48,372 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:53:48,372 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:53:48,376 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:48,376 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. started... 2023-04-12 15:53:48,376 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. as already flushing 2023-04-12 15:53:48,376 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:48,376 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:53:48,376 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:53:48,377 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,377 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:53:48,377 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:53:48,378 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,378 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:48,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:48,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:48,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:48,379 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:48,379 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:53:48,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:48,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:48,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:48,382 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:53:48,382 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@499486df[Count = 0] remaining members to acquire global barrier 2023-04-12 15:53:48,382 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:53:48,382 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:48,384 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:48,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:48,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:48,384 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:53:48,384 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:53:48,384 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:53:48,384 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,384 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:53:48,387 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:53:48,387 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:53:48,387 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:53:48,390 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:48,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:48,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:48,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:48,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:48,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:48,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:48,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:53:48,398 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:53:48,398 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:53:48,399 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:53:48,399 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:53:48,399 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:53:48,399 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:53:48,401 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:48,401 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:53:48,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:48,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:48,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:53:48,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:53:48,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:48,402 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:48,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58374 deadline: 1681314888394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,402 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:48,403 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:48,404 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58398 deadline: 1681314888394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:48,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:48,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:48,404 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:48,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58354 deadline: 1681314888395, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:48,405 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:48,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58340 deadline: 1681314888396, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,406 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:48,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58414 deadline: 1681314888399, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:48,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:48,408 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:48,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:48,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:48,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,411 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:53:48,412 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:53:48,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:53:48,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:53:48,412 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,412 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:48,412 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:48,412 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:53:48,412 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:53:48,412 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:53:48,412 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:53:48,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:53:48,412 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:48,413 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:48,416 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:48,416 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:53:48,416 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:53:48,417 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:53:48,442 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/C/d67524d2cadf4fe2bc51050e662de6be as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/d67524d2cadf4fe2bc51050e662de6be 2023-04-12 15:53:48,452 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in feda8274e3aaad09f0f3a1336e663d20/C of feda8274e3aaad09f0f3a1336e663d20 into d67524d2cadf4fe2bc51050e662de6be(size=12.0 K), total size for store is 23.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:53:48,456 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for feda8274e3aaad09f0f3a1336e663d20: 2023-04-12 15:53:48,456 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20., storeName=feda8274e3aaad09f0f3a1336e663d20/C, priority=12, startTime=1681314827676; duration=0sec 2023-04-12 15:53:48,456 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:53:48,516 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:53:48,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:53:48,520 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:53:48,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:53:48,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:53:48,522 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:53:48,522 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:53:48,523 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:53:48,523 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:53:48,524 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:48,524 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:48,525 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:53:48,525 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:53:48,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:53:48,526 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,526 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:53:48,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:48,527 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:48,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:53:48,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:48,527 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:53:48,527 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:53:48,528 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:53:48,528 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:53:48,528 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:53:48,533 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:53:48,533 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:53:48,533 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:53:48,533 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:48,533 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. started... 2023-04-12 15:53:48,534 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. as already flushing 2023-04-12 15:53:48,534 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:48,534 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:53:48,534 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:53:48,534 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,534 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:53:48,534 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:53:48,539 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:48,539 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:48,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:48,540 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:48,540 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:53:48,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:48,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:48,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:48,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:48,548 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:53:48,548 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@722f35c9[Count = 0] remaining members to acquire global barrier 2023-04-12 15:53:48,548 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:53:48,548 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:48,550 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:48,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:48,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:48,550 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:53:48,551 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:53:48,551 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:53:48,551 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,551 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:53:48,556 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,556 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:53:48,556 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:53:48,556 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:53:48,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:48,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:48,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:48,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:48,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:48,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:48,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:48,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:53:48,564 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:53:48,564 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:53:48,564 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:53:48,564 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:53:48,564 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:53:48,564 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:53:48,566 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:48,566 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:53:48,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:48,566 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:48,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:48,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:48,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:48,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:48,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:53:48,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:48,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:53:48,567 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:48,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:48,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:48,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:48,570 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=357 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/A/0be545459d50431da6044071d99f843d 2023-04-12 15:53:48,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,571 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:48,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:48,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,581 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:53:48,581 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,581 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:53:48,582 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:48,582 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:48,582 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,582 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:48,582 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:48,582 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:48,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:53:48,582 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:53:48,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:53:48,585 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:53:48,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:53:48,585 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:53:48,585 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:53:48,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:53:48,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:53:48,611 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:48,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58340 deadline: 1681314888609, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,612 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:48,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58374 deadline: 1681314888609, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,612 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:48,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58398 deadline: 1681314888609, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,612 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:48,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58414 deadline: 1681314888610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,613 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:48,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58354 deadline: 1681314888611, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,685 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:53:48,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:53:48,689 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:53:48,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:53:48,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:53:48,694 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:53:48,695 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:53:48,696 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:53:48,696 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:53:48,696 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:48,696 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:48,698 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,698 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:53:48,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:53:48,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:53:48,698 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,698 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:53:48,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:48,699 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:48,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:53:48,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:48,700 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:53:48,700 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:53:48,700 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:53:48,700 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:53:48,700 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:53:48,701 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:53:48,712 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:53:48,712 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:53:48,712 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:48,712 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. started... 2023-04-12 15:53:48,712 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. as already flushing 2023-04-12 15:53:48,712 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:48,713 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:53:48,713 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:53:48,713 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,713 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:53:48,713 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:53:48,715 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,715 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:48,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:48,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:48,715 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:48,715 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:53:48,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:48,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:48,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:48,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:48,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:53:48,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29a28032[Count = 0] remaining members to acquire global barrier 2023-04-12 15:53:48,717 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:53:48,717 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:48,719 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:48,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:48,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:48,719 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:53:48,719 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:53:48,719 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:53:48,719 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,719 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:53:48,721 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:48,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:48,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:48,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:48,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:48,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:48,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:48,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:53:48,725 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:53:48,725 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:53:48,726 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:53:48,726 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:53:48,726 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:53:48,726 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:53:48,725 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:53:48,725 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:53:48,726 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:53:48,731 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:48,731 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:53:48,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:48,731 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:48,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:48,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:48,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:48,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:48,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:48,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:53:48,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:53:48,733 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:48,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:48,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:48,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:48,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,735 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:48,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:48,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,739 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,740 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:48,740 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:48,740 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,740 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:48,740 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:48,740 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:48,741 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:53:48,741 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:53:48,741 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:53:48,741 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:53:48,745 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:53:48,745 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:53:48,745 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:53:48,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:53:48,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:53:48,745 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:53:48,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:53:48,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:53:48,845 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:53:48,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:53:48,849 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:53:48,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:53:48,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:53:48,851 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:53:48,852 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:53:48,853 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:53:48,853 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:53:48,853 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:48,853 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:48,854 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,854 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:53:48,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:53:48,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:53:48,855 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,855 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:53:48,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:48,856 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:48,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:53:48,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:48,856 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:53:48,856 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:53:48,857 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:53:48,857 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:53:48,857 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:53:48,858 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:53:48,868 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:53:48,868 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:53:48,868 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:48,868 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. started... 2023-04-12 15:53:48,868 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. as already flushing 2023-04-12 15:53:48,869 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:48,869 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:53:48,869 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:53:48,869 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,869 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:53:48,869 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:53:48,871 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,871 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:48,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:48,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:48,872 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:48,872 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:53:48,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:48,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:48,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:48,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:48,875 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:53:48,875 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a675501[Count = 0] remaining members to acquire global barrier 2023-04-12 15:53:48,875 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:53:48,876 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:48,877 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:48,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:48,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:48,877 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:53:48,877 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:53:48,877 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:53:48,878 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,878 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:53:48,879 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:53:48,879 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,879 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:53:48,879 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:53:48,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:48,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:48,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:48,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:48,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:48,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:48,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:48,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:53:48,883 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:53:48,883 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:53:48,883 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:53:48,883 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:53:48,883 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:53:48,883 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:53:48,885 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:48,885 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:53:48,885 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:48,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:48,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:48,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:48,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:48,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:48,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:53:48,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:53:48,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:48,886 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:48,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:48,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:48,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:48,888 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:48,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:48,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,912 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:53:48,912 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:53:48,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:53:48,912 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,912 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:48,912 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:53:48,912 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:53:48,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:53:48,912 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:48,912 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,912 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:48,913 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:48,913 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:53:48,913 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:53:48,912 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:53:48,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:53:48,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:53:48,913 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:48,918 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:48,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58398 deadline: 1681314888915, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,919 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:48,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58340 deadline: 1681314888917, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,920 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:48,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58414 deadline: 1681314888917, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,921 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:48,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58374 deadline: 1681314888918, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:48,921 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:48,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58354 deadline: 1681314888918, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:49,013 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:53:49,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:53:49,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:53:49,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:53:49,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:53:49,018 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:53:49,018 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:53:49,021 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:53:49,021 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:53:49,023 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:49,023 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:49,025 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:49,025 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:53:49,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:53:49,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:53:49,026 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=357 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/B/7ee6b66de8594674a282cac1c1a4734e 2023-04-12 15:53:49,026 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:49,026 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:53:49,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:49,027 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:49,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:53:49,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:49,027 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:53:49,027 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:53:49,028 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:53:49,028 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:53:49,029 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:53:49,031 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:53:49,031 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:53:49,031 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:49,031 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. started... 2023-04-12 15:53:49,031 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. as already flushing 2023-04-12 15:53:49,031 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:49,031 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:53:49,031 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:53:49,031 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:49,031 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:53:49,031 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:53:49,034 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:49,034 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:49,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:49,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:49,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:49,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:49,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:49,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:49,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:49,037 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:49,037 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:53:49,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:49,038 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:53:49,038 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77289aec[Count = 0] remaining members to acquire global barrier 2023-04-12 15:53:49,038 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:53:49,038 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:49,043 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:49,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:49,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:49,044 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:53:49,044 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:53:49,044 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:53:49,044 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:49,044 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:53:49,046 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:49,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:49,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:49,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:49,047 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:53:49,047 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:53:49,047 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:53:49,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:49,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:49,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:49,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:49,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:49,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:49,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:49,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:53:49,085 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:53:49,085 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:53:49,085 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:53:49,085 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:53:49,085 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:53:49,085 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:53:49,088 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:49,088 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:49,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:49,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:49,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:49,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:49,088 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:53:49,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:49,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:49,089 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:49,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:53:49,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:53:49,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:49,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:49,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:49,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:49,091 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:49,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:49,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:49,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:49,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:49,095 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:49,095 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:53:49,095 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:53:49,095 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:53:49,095 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:53:49,095 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:49,095 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:49,095 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:49,095 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:49,095 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:49,095 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:49,095 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:53:49,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:53:49,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:53:49,095 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:53:49,101 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:53:49,102 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:53:49,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:53:49,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:53:49,202 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:53:49,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:53:49,206 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:53:49,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:53:49,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:53:49,210 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:53:49,210 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:53:49,211 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:53:49,212 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:53:49,212 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:49,212 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:49,215 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:53:49,215 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:49,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:53:49,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:53:49,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:49,216 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:49,216 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:53:49,218 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:49,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:53:49,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:49,219 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:53:49,219 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:53:49,219 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:53:49,219 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:53:49,219 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:53:49,220 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:53:49,221 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:53:49,221 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:53:49,221 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:49,221 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. started... 2023-04-12 15:53:49,221 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. as already flushing 2023-04-12 15:53:49,222 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:49,222 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:53:49,222 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:53:49,222 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:49,222 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:53:49,222 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:53:49,224 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:49,224 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:49,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:49,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:49,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:49,228 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:49,228 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:53:49,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:49,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:49,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:49,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:49,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:49,237 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:53:49,237 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ca268bd[Count = 0] remaining members to acquire global barrier 2023-04-12 15:53:49,237 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:53:49,238 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:49,239 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:49,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:49,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:49,240 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:53:49,240 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:53:49,240 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:53:49,240 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:49,240 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:53:49,241 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:53:49,242 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:53:49,241 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:49,242 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:53:49,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:49,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:49,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:49,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:49,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:49,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:49,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:49,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:49,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:49,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:49,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:53:49,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:53:49,254 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:53:49,255 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:53:49,255 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:53:49,255 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:53:49,255 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:53:49,270 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:49,270 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:53:49,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:49,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:49,270 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:49,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:49,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:49,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:49,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:49,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:53:49,272 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:49,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:53:49,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:49,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:49,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:49,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:49,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:49,280 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:49,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:49,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:49,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:49,286 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:49,286 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:53:49,286 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:53:49,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:53:49,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:53:49,286 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:49,286 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:53:49,287 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:53:49,286 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:53:49,287 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:49,288 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:49,288 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:49,288 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:49,288 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:53:49,288 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:53:49,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:53:49,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:53:49,288 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:49,388 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:53:49,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:53:49,391 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:53:49,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:53:49,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:53:49,394 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:53:49,394 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:53:49,395 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:53:49,395 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:53:49,395 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:49,395 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:49,397 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:53:49,397 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:49,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:53:49,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:53:49,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:49,398 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:49,398 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:49,398 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:53:49,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:53:49,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:49,404 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:53:49,407 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:53:49,407 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:53:49,407 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:53:49,407 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:53:49,408 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:53:49,408 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:53:49,408 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:53:49,408 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:49,409 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. started... 2023-04-12 15:53:49,409 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. as already flushing 2023-04-12 15:53:49,409 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:49,409 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:53:49,409 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:53:49,409 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:49,409 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:53:49,409 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:53:49,411 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:49,411 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:49,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:49,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:49,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:49,411 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:49,411 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:53:49,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:49,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:49,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:49,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:49,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:49,416 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:53:49,416 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41bcd40b[Count = 0] remaining members to acquire global barrier 2023-04-12 15:53:49,416 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:53:49,416 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:49,423 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:49,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:49,423 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:49,424 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:53:49,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:49,424 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:53:49,424 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:53:49,424 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:53:49,425 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:49,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58340 deadline: 1681314889423, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:49,426 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:49,426 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:53:49,426 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:49,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58414 deadline: 1681314889424, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:49,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:49,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:49,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:49,426 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:49,426 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:53:49,427 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:49,427 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:53:49,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:49,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58398 deadline: 1681314889425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:49,428 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:49,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:49,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:49,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:49,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:49,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:49,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:49,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58374 deadline: 1681314889425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:49,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:53:49,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:53:49,433 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:53:49,433 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:53:49,433 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:53:49,433 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:53:49,433 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:53:49,436 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:49,436 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:53:49,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:49,436 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:49,436 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:49,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:49,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:49,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:49,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:49,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:53:49,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:53:49,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:49,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:49,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:49,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:49,439 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:49,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:49,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:49,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:49,442 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:53:49,443 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:53:49,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:53:49,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:53:49,443 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:49,443 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:49,443 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:49,443 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:49,443 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:49,443 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:53:49,443 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:49,443 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:49,443 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:53:49,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:53:49,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58354 deadline: 1681314889425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:49,445 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:53:49,445 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:53:49,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:53:49,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:53:49,447 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:53:49,504 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=357 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/C/59ad81992fef445b846da0c27cb3b37e 2023-04-12 15:53:49,519 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/A/0be545459d50431da6044071d99f843d as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/0be545459d50431da6044071d99f843d 2023-04-12 15:53:49,529 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/0be545459d50431da6044071d99f843d, entries=250, sequenceid=357, filesize=16.7 K 2023-04-12 15:53:49,533 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/B/7ee6b66de8594674a282cac1c1a4734e as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/7ee6b66de8594674a282cac1c1a4734e 2023-04-12 15:53:49,545 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:53:49,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:53:49,549 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/7ee6b66de8594674a282cac1c1a4734e, entries=150, sequenceid=357, filesize=11.9 K 2023-04-12 15:53:49,553 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/C/59ad81992fef445b846da0c27cb3b37e as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/59ad81992fef445b846da0c27cb3b37e 2023-04-12 15:53:49,553 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:53:49,563 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/59ad81992fef445b846da0c27cb3b37e, entries=150, sequenceid=357, filesize=11.9 K 2023-04-12 15:53:49,565 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~86.11 KB/88176, currentSize=161.02 KB/164880 for feda8274e3aaad09f0f3a1336e663d20 in 1422ms, sequenceid=357, compaction requested=true 2023-04-12 15:53:49,565 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for feda8274e3aaad09f0f3a1336e663d20: 2023-04-12 15:53:49,565 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:53:49,565 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 15:53:49,565 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 15:53:49,565 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:53:49,566 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:53:49,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:53:49,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:53:49,571 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:53:49,571 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 41576 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 15:53:49,571 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:53:49,571 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1912): feda8274e3aaad09f0f3a1336e663d20/A is initiating minor compaction (all files) 2023-04-12 15:53:49,571 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of feda8274e3aaad09f0f3a1336e663d20/A in TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:49,571 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36696 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 15:53:49,572 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/dce1631ad0724f95b7b4c1386ccfdaa9, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/6c9aeb7cb4bc41d3b501757541eb2baf, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/0be545459d50431da6044071d99f843d] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp, totalSize=40.6 K 2023-04-12 15:53:49,572 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1912): feda8274e3aaad09f0f3a1336e663d20/B is initiating minor compaction (all files) 2023-04-12 15:53:49,572 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2259): Starting compaction of feda8274e3aaad09f0f3a1336e663d20/B in TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:49,572 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/59cfd2bedaaa4e0688bb8f241c4d75d3, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/73fd13d4bc004b969365cce1d296fbb4, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/7ee6b66de8594674a282cac1c1a4734e] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp, totalSize=35.8 K 2023-04-12 15:53:49,572 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting dce1631ad0724f95b7b4c1386ccfdaa9, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=320, earliestPutTs=1681314826375 2023-04-12 15:53:49,577 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:53:49,573 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 59cfd2bedaaa4e0688bb8f241c4d75d3, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=320, earliestPutTs=1681314826375 2023-04-12 15:53:49,578 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:53:49,579 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 6c9aeb7cb4bc41d3b501757541eb2baf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=334, earliestPutTs=1681314826479 2023-04-12 15:53:49,579 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:49,579 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:49,579 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 73fd13d4bc004b969365cce1d296fbb4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=334, earliestPutTs=1681314826479 2023-04-12 15:53:49,580 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 7ee6b66de8594674a282cac1c1a4734e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=357, earliestPutTs=1681314827762 2023-04-12 15:53:49,580 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 0be545459d50431da6044071d99f843d, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=357, earliestPutTs=1681314827762 2023-04-12 15:53:49,581 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:53:49,581 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:49,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:53:49,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:53:49,582 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:49,582 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:53:49,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:49,582 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:49,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:53:49,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:49,583 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:53:49,588 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:53:49,588 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:53:49,588 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:53:49,588 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:53:49,593 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:53:49,605 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:53:49,605 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:53:49,605 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:49,606 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. started... 2023-04-12 15:53:49,606 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing feda8274e3aaad09f0f3a1336e663d20 3/3 column families, dataSize=161.02 KB heapSize=445.92 KB 2023-04-12 15:53:49,606 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK feda8274e3aaad09f0f3a1336e663d20, store=A 2023-04-12 15:53:49,606 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:53:49,606 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK feda8274e3aaad09f0f3a1336e663d20, store=B 2023-04-12 15:53:49,607 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:53:49,607 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK feda8274e3aaad09f0f3a1336e663d20, store=C 2023-04-12 15:53:49,607 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:53:49,629 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] throttle.PressureAwareThroughputController(145): feda8274e3aaad09f0f3a1336e663d20#B#compaction#21 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:53:49,629 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] throttle.PressureAwareThroughputController(145): feda8274e3aaad09f0f3a1336e663d20#A#compaction#22 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:53:49,667 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=396 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/A/c782134cb55b47929b5e8647db87f2bc 2023-04-12 15:53:49,685 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/B/9ddf29945a2747a7a51787356b0ca7d9 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/9ddf29945a2747a7a51787356b0ca7d9 2023-04-12 15:53:49,689 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/A/c2497c9e947b4049aa03fa5ba5d8741e as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/c2497c9e947b4049aa03fa5ba5d8741e 2023-04-12 15:53:49,702 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in feda8274e3aaad09f0f3a1336e663d20/B of feda8274e3aaad09f0f3a1336e663d20 into 9ddf29945a2747a7a51787356b0ca7d9(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:53:49,702 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for feda8274e3aaad09f0f3a1336e663d20: 2023-04-12 15:53:49,702 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20., storeName=feda8274e3aaad09f0f3a1336e663d20/B, priority=13, startTime=1681314829565; duration=0sec 2023-04-12 15:53:49,702 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in feda8274e3aaad09f0f3a1336e663d20/A of feda8274e3aaad09f0f3a1336e663d20 into c2497c9e947b4049aa03fa5ba5d8741e(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:53:49,702 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for feda8274e3aaad09f0f3a1336e663d20: 2023-04-12 15:53:49,702 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20., storeName=feda8274e3aaad09f0f3a1336e663d20/A, priority=13, startTime=1681314829565; duration=0sec 2023-04-12 15:53:49,704 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:53:49,704 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:53:49,704 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 15:53:49,708 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36696 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 15:53:49,709 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1912): feda8274e3aaad09f0f3a1336e663d20/C is initiating minor compaction (all files) 2023-04-12 15:53:49,709 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2259): Starting compaction of feda8274e3aaad09f0f3a1336e663d20/C in TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:49,709 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/d67524d2cadf4fe2bc51050e662de6be, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/bb83c8786ab54e3fb162fac4fe55e697, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/59ad81992fef445b846da0c27cb3b37e] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp, totalSize=35.8 K 2023-04-12 15:53:49,710 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting d67524d2cadf4fe2bc51050e662de6be, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=320, earliestPutTs=1681314826375 2023-04-12 15:53:49,711 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting bb83c8786ab54e3fb162fac4fe55e697, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=334, earliestPutTs=1681314826479 2023-04-12 15:53:49,712 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 59ad81992fef445b846da0c27cb3b37e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=357, earliestPutTs=1681314827762 2023-04-12 15:53:49,762 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] throttle.PressureAwareThroughputController(145): feda8274e3aaad09f0f3a1336e663d20#C#compaction#25 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:53:49,791 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/C/c51630a3e68945b1bf5b2cdfd8408313 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/c51630a3e68945b1bf5b2cdfd8408313 2023-04-12 15:53:49,804 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in feda8274e3aaad09f0f3a1336e663d20/C of feda8274e3aaad09f0f3a1336e663d20 into c51630a3e68945b1bf5b2cdfd8408313(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:53:49,804 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for feda8274e3aaad09f0f3a1336e663d20: 2023-04-12 15:53:49,804 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20., storeName=feda8274e3aaad09f0f3a1336e663d20/C, priority=13, startTime=1681314829566; duration=0sec 2023-04-12 15:53:49,805 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:53:50,134 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=396 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/B/8697c6e31a084364b7a5086786027a9b 2023-04-12 15:53:50,430 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. as already flushing 2023-04-12 15:53:50,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on feda8274e3aaad09f0f3a1336e663d20 2023-04-12 15:53:50,439 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:50,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58414 deadline: 1681314890439, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:50,445 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:50,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58374 deadline: 1681314890443, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:50,449 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:50,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58340 deadline: 1681314890448, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:50,449 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:50,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58398 deadline: 1681314890449, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:50,460 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:50,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58354 deadline: 1681314890457, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:50,552 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:50,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58414 deadline: 1681314890550, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:50,553 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:50,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58340 deadline: 1681314890551, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:50,578 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=396 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/C/75f0e5fa8f42449aa8707c2fb0b7ae2a 2023-04-12 15:53:50,590 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/A/c782134cb55b47929b5e8647db87f2bc as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/c782134cb55b47929b5e8647db87f2bc 2023-04-12 15:53:50,597 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/c782134cb55b47929b5e8647db87f2bc, entries=150, sequenceid=396, filesize=11.9 K 2023-04-12 15:53:50,600 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/B/8697c6e31a084364b7a5086786027a9b as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/8697c6e31a084364b7a5086786027a9b 2023-04-12 15:53:50,607 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/8697c6e31a084364b7a5086786027a9b, entries=150, sequenceid=396, filesize=11.9 K 2023-04-12 15:53:50,609 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/C/75f0e5fa8f42449aa8707c2fb0b7ae2a as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/75f0e5fa8f42449aa8707c2fb0b7ae2a 2023-04-12 15:53:50,615 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/75f0e5fa8f42449aa8707c2fb0b7ae2a, entries=150, sequenceid=396, filesize=11.9 K 2023-04-12 15:53:50,617 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~161.02 KB/164880, heapSize ~445.88 KB/456576, currentSize=26.84 KB/27480 for feda8274e3aaad09f0f3a1336e663d20 in 1011ms, sequenceid=396, compaction requested=false 2023-04-12 15:53:50,617 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for feda8274e3aaad09f0f3a1336e663d20: 2023-04-12 15:53:50,617 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:50,617 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:53:50,618 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:53:50,618 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:50,618 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:53:50,619 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:53:50,620 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:50,620 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:50,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:50,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:50,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:50,621 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:50,621 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:53:50,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:50,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:50,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:50,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:50,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:50,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:53:50,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61d35e60[Count = 0] remaining members to acquire global barrier 2023-04-12 15:53:50,623 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:53:50,623 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:50,624 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:50,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:50,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:50,624 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:53:50,624 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:53:50,624 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:53:50,625 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:50,625 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:53:50,626 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:50,626 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:53:50,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:50,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:50,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:50,626 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:53:50,626 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:53:50,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:50,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:50,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:50,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:50,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:50,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:50,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:50,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:53:50,631 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:53:50,631 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:53:50,631 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:53:50,631 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:53:50,631 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:53:50,631 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:53:50,643 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:50,643 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:50,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:50,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:50,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:50,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:50,643 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:53:50,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:50,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:50,644 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:50,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:50,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:53:50,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:53:50,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:50,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:50,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:50,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:50,645 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:50,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:50,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:50,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:50,649 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:50,649 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:53:50,649 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:50,649 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:53:50,649 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:50,649 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:53:50,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:53:50,649 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:53:50,650 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:53:50,649 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:50,650 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:50,650 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:53:50,650 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:53:50,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:53:50,650 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:50,651 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:50,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:53:50,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:53:50,751 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:53:50,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:53:50,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on feda8274e3aaad09f0f3a1336e663d20 2023-04-12 15:53:50,761 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing feda8274e3aaad09f0f3a1336e663d20 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-12 15:53:50,762 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK feda8274e3aaad09f0f3a1336e663d20, store=A 2023-04-12 15:53:50,762 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:53:50,762 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK feda8274e3aaad09f0f3a1336e663d20, store=B 2023-04-12 15:53:50,762 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:53:50,762 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK feda8274e3aaad09f0f3a1336e663d20, store=C 2023-04-12 15:53:50,762 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:53:50,769 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:53:50,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:53:50,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:53:50,781 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:53:50,781 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:53:50,788 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:53:50,789 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:53:50,789 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:50,789 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:50,791 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:53:50,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:53:50,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:53:50,791 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:50,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:50,793 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:50,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:53:50,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:50,794 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:53:50,800 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:53:50,800 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:53:50,802 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:50,803 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:53:50,803 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:53:50,804 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:53:50,805 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:53:50,805 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:53:50,805 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:53:50,806 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:50,806 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. started... 2023-04-12 15:53:50,806 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. as already flushing 2023-04-12 15:53:50,806 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:50,806 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:53:50,806 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:53:50,806 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:50,806 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:53:50,806 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:53:50,809 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:50,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:50,809 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:50,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:50,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:50,809 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:50,809 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:53:50,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:50,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:50,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:50,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:50,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:50,812 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:53:50,812 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@521f574c[Count = 0] remaining members to acquire global barrier 2023-04-12 15:53:50,812 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:53:50,812 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:50,814 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:50,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:50,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:50,815 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:53:50,815 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:53:50,815 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:53:50,815 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:50,815 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:53:50,816 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:50,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:50,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:50,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:50,817 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=409 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/A/74dac36decc3418b95770ac6d31bdfaa 2023-04-12 15:53:50,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:50,818 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:53:50,818 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:53:50,818 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:53:50,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:50,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:50,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:50,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:50,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:50,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:50,824 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681314830792, max timestamp=1681314830818Num uniques 859; 2023-04-12 15:53:50,824 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-12 15:53:50,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:53:50,824 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:53:50,824 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:53:50,824 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:53:50,825 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:53:50,825 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:53:50,825 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:53:50,826 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681314830792, max timestamp=1681314830818Num uniques 859; 2023-04-12 15:53:50,826 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-12 15:53:50,831 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681314830792, max timestamp=1681314830818Num uniques 859; 2023-04-12 15:53:50,831 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-12 15:53:50,832 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:50,832 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:50,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:50,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:50,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:50,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:50,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:50,832 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:53:50,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:53:50,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:53:50,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:50,833 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:50,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:50,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:50,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:50,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:50,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:50,840 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:50,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:50,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:50,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:50,845 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:53:50,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:53:50,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:53:50,845 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:53:50,845 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:50,846 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:50,846 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:50,846 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:50,846 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:50,846 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:50,846 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:50,845 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:53:50,847 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:53:50,845 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:53:50,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:53:50,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:53:50,848 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:53:50,848 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:53:50,856 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=409 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/B/039c36f07b6749e0a3caef35b4b8abc0 2023-04-12 15:53:50,882 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=409 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/C/6312d05681f54994b07b846ccae53c44 2023-04-12 15:53:50,890 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/A/74dac36decc3418b95770ac6d31bdfaa as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/74dac36decc3418b95770ac6d31bdfaa 2023-04-12 15:53:50,892 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681314830838, max timestamp=1681314830890Num uniques 1018; 2023-04-12 15:53:50,892 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-12 15:53:50,893 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681314830838, max timestamp=1681314830890Num uniques 1068; 2023-04-12 15:53:50,893 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681314830838, max timestamp=1681314830890Num uniques 1018; 2023-04-12 15:53:50,893 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-04-12 15:53:50,893 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-12 15:53:50,898 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/74dac36decc3418b95770ac6d31bdfaa, entries=250, sequenceid=409, filesize=16.7 K 2023-04-12 15:53:50,900 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/B/039c36f07b6749e0a3caef35b4b8abc0 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/039c36f07b6749e0a3caef35b4b8abc0 2023-04-12 15:53:50,908 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/039c36f07b6749e0a3caef35b4b8abc0, entries=150, sequenceid=409, filesize=11.9 K 2023-04-12 15:53:50,910 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/C/6312d05681f54994b07b846ccae53c44 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/6312d05681f54994b07b846ccae53c44 2023-04-12 15:53:50,917 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/6312d05681f54994b07b846ccae53c44, entries=150, sequenceid=409, filesize=11.9 K 2023-04-12 15:53:50,920 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=87.22 KB/89310 for feda8274e3aaad09f0f3a1336e663d20 in 159ms, sequenceid=409, compaction requested=true 2023-04-12 15:53:50,920 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for feda8274e3aaad09f0f3a1336e663d20: 2023-04-12 15:53:50,920 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:53:50,920 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 15:53:50,920 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 15:53:50,920 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:53:50,921 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:53:50,922 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 41678 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 15:53:50,922 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1912): feda8274e3aaad09f0f3a1336e663d20/A is initiating minor compaction (all files) 2023-04-12 15:53:50,922 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36798 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 15:53:50,922 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of feda8274e3aaad09f0f3a1336e663d20/A in TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:50,922 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1912): feda8274e3aaad09f0f3a1336e663d20/B is initiating minor compaction (all files) 2023-04-12 15:53:50,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on feda8274e3aaad09f0f3a1336e663d20 2023-04-12 15:53:50,922 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2259): Starting compaction of feda8274e3aaad09f0f3a1336e663d20/B in TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:50,923 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/9ddf29945a2747a7a51787356b0ca7d9, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/8697c6e31a084364b7a5086786027a9b, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/039c36f07b6749e0a3caef35b4b8abc0] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp, totalSize=35.9 K 2023-04-12 15:53:50,922 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/c2497c9e947b4049aa03fa5ba5d8741e, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/c782134cb55b47929b5e8647db87f2bc, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/74dac36decc3418b95770ac6d31bdfaa] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp, totalSize=40.7 K 2023-04-12 15:53:50,923 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 9ddf29945a2747a7a51787356b0ca7d9, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=357, earliestPutTs=1681314827762 2023-04-12 15:53:50,924 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing feda8274e3aaad09f0f3a1336e663d20 3/3 column families, dataSize=100.63 KB heapSize=275.16 KB 2023-04-12 15:53:50,924 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting c2497c9e947b4049aa03fa5ba5d8741e, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=357, earliestPutTs=1681314827762 2023-04-12 15:53:50,925 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK feda8274e3aaad09f0f3a1336e663d20, store=A 2023-04-12 15:53:50,925 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 8697c6e31a084364b7a5086786027a9b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=396, earliestPutTs=1681314828213 2023-04-12 15:53:50,925 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:53:50,926 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK feda8274e3aaad09f0f3a1336e663d20, store=B 2023-04-12 15:53:50,926 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:53:50,926 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK feda8274e3aaad09f0f3a1336e663d20, store=C 2023-04-12 15:53:50,926 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting c782134cb55b47929b5e8647db87f2bc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=396, earliestPutTs=1681314828213 2023-04-12 15:53:50,926 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:53:50,926 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 039c36f07b6749e0a3caef35b4b8abc0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=409, earliestPutTs=1681314830435 2023-04-12 15:53:50,927 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 74dac36decc3418b95770ac6d31bdfaa, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=409, earliestPutTs=1681314830435 2023-04-12 15:53:50,945 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] throttle.PressureAwareThroughputController(145): feda8274e3aaad09f0f3a1336e663d20#A#compaction#31 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:53:50,948 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:53:50,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:53:50,953 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:53:50,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:53:50,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:53:50,960 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:53:50,960 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:53:50,962 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:53:50,962 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:53:50,963 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:50,963 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:50,963 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] throttle.PressureAwareThroughputController(145): feda8274e3aaad09f0f3a1336e663d20#B#compaction#32 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:53:50,965 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:50,966 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:50,966 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:53:50,973 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:53:50,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:53:50,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:53:50,976 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:50,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58340 deadline: 1681314890966, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:50,977 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:50,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58414 deadline: 1681314890966, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:50,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:50,978 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:50,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:53:50,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:50,979 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:53:50,984 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:53:50,984 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=456 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/A/c2792699c7db48388370f7b2ee7057ef 2023-04-12 15:53:50,984 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:53:50,984 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:53:50,984 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:53:50,986 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:53:50,988 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:53:50,988 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:53:50,988 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:50,988 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. started... 2023-04-12 15:53:50,988 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. as already flushing 2023-04-12 15:53:50,988 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:50,988 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:53:50,988 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:53:50,988 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:50,989 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:53:50,989 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:53:50,990 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:50,990 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:50,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:50,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:50,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:50,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:50,991 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:50,991 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:53:50,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:50,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:50,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:50,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:50,994 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:53:50,994 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d973a32[Count = 0] remaining members to acquire global barrier 2023-04-12 15:53:50,995 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:53:50,995 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:50,997 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:50,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:50,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:50,997 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:53:50,997 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:53:50,997 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:53:50,998 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:50,998 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:53:50,999 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/A/96e5ebbbdecd47f4a574785d8f4e4ea2 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/96e5ebbbdecd47f4a574785d8f4e4ea2 2023-04-12 15:53:51,000 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:51,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:51,000 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:53:51,000 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:53:51,000 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:53:51,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:51,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:51,000 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/B/fe8d0f5e333c407d9cbd1335556a698b as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/fe8d0f5e333c407d9cbd1335556a698b 2023-04-12 15:53:51,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:51,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:51,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:51,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:53:51,003 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:53:51,003 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:53:51,003 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:53:51,003 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:53:51,003 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:53:51,004 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:53:51,008 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:51,008 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:51,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:51,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:51,008 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:53:51,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:51,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:51,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:53:51,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:53:51,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:51,009 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in feda8274e3aaad09f0f3a1336e663d20/A of feda8274e3aaad09f0f3a1336e663d20 into 96e5ebbbdecd47f4a574785d8f4e4ea2(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:53:51,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:51,009 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for feda8274e3aaad09f0f3a1336e663d20: 2023-04-12 15:53:51,009 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,010 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20., storeName=feda8274e3aaad09f0f3a1336e663d20/A, priority=13, startTime=1681314830920; duration=0sec 2023-04-12 15:53:51,010 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:53:51,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:51,010 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 15:53:51,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:51,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:51,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:51,013 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36798 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 15:53:51,013 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1912): feda8274e3aaad09f0f3a1336e663d20/C is initiating minor compaction (all files) 2023-04-12 15:53:51,013 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of feda8274e3aaad09f0f3a1336e663d20/C in TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:51,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,013 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/c51630a3e68945b1bf5b2cdfd8408313, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/75f0e5fa8f42449aa8707c2fb0b7ae2a, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/6312d05681f54994b07b846ccae53c44] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp, totalSize=35.9 K 2023-04-12 15:53:51,013 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,014 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting c51630a3e68945b1bf5b2cdfd8408313, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=357, earliestPutTs=1681314827762 2023-04-12 15:53:51,018 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 75f0e5fa8f42449aa8707c2fb0b7ae2a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=396, earliestPutTs=1681314828213 2023-04-12 15:53:51,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:51,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:51,021 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 6312d05681f54994b07b846ccae53c44, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=409, earliestPutTs=1681314830435 2023-04-12 15:53:51,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,023 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in feda8274e3aaad09f0f3a1336e663d20/B of feda8274e3aaad09f0f3a1336e663d20 into fe8d0f5e333c407d9cbd1335556a698b(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:53:51,023 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for feda8274e3aaad09f0f3a1336e663d20: 2023-04-12 15:53:51,023 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20., storeName=feda8274e3aaad09f0f3a1336e663d20/B, priority=13, startTime=1681314830920; duration=0sec 2023-04-12 15:53:51,023 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:53:51,026 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,026 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:51,026 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:53:51,026 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:53:51,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:53:51,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:53:51,026 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:53:51,026 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:51,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:53:51,026 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:53:51,032 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:53:51,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:53:51,032 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,032 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:51,032 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:51,032 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:51,032 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:53:51,032 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:53:51,041 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] throttle.PressureAwareThroughputController(145): feda8274e3aaad09f0f3a1336e663d20#C#compaction#34 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:53:51,082 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:51,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58414 deadline: 1681314891078, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,083 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:51,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58340 deadline: 1681314891078, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,133 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:53:51,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:53:51,136 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:53:51,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:53:51,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:53:51,138 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:53:51,138 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:53:51,138 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:53:51,138 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:53:51,138 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:51,138 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:51,140 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,140 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:53:51,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:53:51,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:53:51,140 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,140 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:53:51,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:51,141 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:51,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:53:51,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:51,141 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:53:51,142 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:53:51,142 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:53:51,142 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:53:51,142 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:53:51,142 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:53:51,143 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:53:51,143 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:53:51,143 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:51,143 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. started... 2023-04-12 15:53:51,143 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. as already flushing 2023-04-12 15:53:51,143 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:51,144 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:53:51,144 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:53:51,144 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,144 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:53:51,144 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:53:51,146 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,146 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:51,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:51,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:51,146 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:51,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:51,146 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:53:51,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:51,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:51,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:51,148 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:53:51,148 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1aacad07[Count = 0] remaining members to acquire global barrier 2023-04-12 15:53:51,148 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:53:51,148 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:51,151 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:51,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:51,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:51,151 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:53:51,151 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:53:51,151 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:53:51,151 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,151 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:53:51,153 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:53:51,153 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:53:51,153 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,153 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:53:51,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:51,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:51,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:51,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:51,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:51,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:51,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:51,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:53:51,156 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:53:51,156 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:53:51,157 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:53:51,157 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:53:51,157 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:53:51,157 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:53:51,158 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:51,158 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:51,158 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:53:51,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:51,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:51,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:51,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:51,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:51,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:53:51,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:53:51,159 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:51,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:51,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:51,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:51,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:51,161 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:51,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:51,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,172 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,172 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:51,172 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:51,172 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:53:51,172 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:53:51,172 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:53:51,172 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:53:51,172 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,172 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:53:51,172 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:53:51,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:53:51,172 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:51,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:53:51,172 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:51,172 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:53:51,173 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:53:51,172 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:51,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:53:51,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:53:51,273 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:53:51,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:53:51,276 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:53:51,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:53:51,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:53:51,278 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:53:51,278 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:53:51,278 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:53:51,278 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:53:51,278 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:51,278 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:51,280 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:53:51,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:53:51,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:53:51,280 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:51,280 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,280 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:53:51,281 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:51,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:53:51,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:51,281 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:53:51,285 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:51,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58340 deadline: 1681314891284, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,286 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:53:51,286 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:53:51,286 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:53:51,286 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:53:51,286 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:53:51,286 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:53:51,286 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:53:51,286 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:51,286 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. started... 2023-04-12 15:53:51,286 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. as already flushing 2023-04-12 15:53:51,286 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:51,286 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:53:51,286 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:53:51,286 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,286 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:53:51,287 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:53:51,288 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:51,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58414 deadline: 1681314891285, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,297 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,297 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:51,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:51,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:51,297 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:51,297 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:53:51,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:51,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:51,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:51,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:51,299 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:53:51,299 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6177d06c[Count = 0] remaining members to acquire global barrier 2023-04-12 15:53:51,299 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:53:51,299 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:51,301 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:51,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:51,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:51,301 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:53:51,301 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:53:51,301 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:53:51,301 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,301 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:53:51,304 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:53:51,304 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,304 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:53:51,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:51,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:51,304 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:53:51,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:51,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:51,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:51,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:51,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:51,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:53:51,307 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:53:51,307 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:53:51,307 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:53:51,307 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:53:51,307 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:53:51,307 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:53:51,309 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:51,309 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:51,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:51,309 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:53:51,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:51,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:51,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:51,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:51,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:53:51,310 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:51,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:53:51,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:51,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:51,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:51,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:51,311 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:51,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:51,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,314 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,314 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:51,314 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:51,314 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:53:51,314 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:53:51,314 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:53:51,314 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:51,314 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:51,314 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:51,314 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:53:51,314 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:53:51,314 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:53:51,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:53:51,315 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:53:51,315 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:53:51,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:53:51,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:53:51,415 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:53:51,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:53:51,418 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:53:51,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:53:51,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:53:51,419 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:53:51,419 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:53:51,419 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:53:51,420 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:53:51,420 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:51,420 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:51,421 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,421 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:53:51,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:53:51,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:53:51,422 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,422 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:53:51,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:51,422 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:51,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:53:51,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:51,423 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:53:51,423 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:53:51,423 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:53:51,423 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:53:51,423 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:53:51,423 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:53:51,423 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:53:51,423 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:53:51,423 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:51,424 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. started... 2023-04-12 15:53:51,424 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. as already flushing 2023-04-12 15:53:51,424 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:51,424 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:53:51,424 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:53:51,424 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,424 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:53:51,424 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:53:51,427 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,427 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:51,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:51,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:51,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:51,428 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:51,428 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:53:51,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:51,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:51,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:51,429 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:53:51,429 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@497a5e39[Count = 0] remaining members to acquire global barrier 2023-04-12 15:53:51,429 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:53:51,429 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:51,430 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:51,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:51,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:51,431 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:53:51,431 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:53:51,431 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,431 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:53:51,431 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:53:51,432 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,432 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:53:51,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:51,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:51,433 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:53:51,433 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:53:51,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:51,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:51,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:51,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:51,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:51,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:53:51,435 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:53:51,435 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:53:51,435 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:53:51,435 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:53:51,435 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:53:51,435 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:53:51,437 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:51,437 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:53:51,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:51,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:51,438 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:51,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:51,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:51,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:51,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:53:51,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:53:51,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:51,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:51,440 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:51,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:51,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:51,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,441 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:51,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:51,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,445 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:53:51,445 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:53:51,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:53:51,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:53:51,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:53:51,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:53:51,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:53:51,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:53:51,445 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:53:51,445 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:53:51,445 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:53:51,447 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,447 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:51,447 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:51,447 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,447 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:51,447 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:51,447 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:51,450 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=456 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/B/03499d929f184829aefcaafcd7ef27f5 2023-04-12 15:53:51,470 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/C/10feb66066be4a55bb8f1a2f2998be04 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/10feb66066be4a55bb8f1a2f2998be04 2023-04-12 15:53:51,478 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in feda8274e3aaad09f0f3a1336e663d20/C of feda8274e3aaad09f0f3a1336e663d20 into 10feb66066be4a55bb8f1a2f2998be04(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:53:51,478 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for feda8274e3aaad09f0f3a1336e663d20: 2023-04-12 15:53:51,478 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20., storeName=feda8274e3aaad09f0f3a1336e663d20/C, priority=13, startTime=1681314830921; duration=0sec 2023-04-12 15:53:51,479 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:53:51,546 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:53:51,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:53:51,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:53:51,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:53:51,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:53:51,551 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:53:51,551 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:53:51,551 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:53:51,551 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:53:51,552 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:51,552 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:51,553 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:53:51,553 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:53:51,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:53:51,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:51,554 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,554 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:53:51,555 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:51,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:53:51,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:51,555 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:53:51,556 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:53:51,556 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:53:51,556 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:53:51,556 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:53:51,556 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:53:51,556 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:53:51,556 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:53:51,556 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:51,556 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. started... 2023-04-12 15:53:51,556 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. as already flushing 2023-04-12 15:53:51,556 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:51,556 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:53:51,557 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:53:51,557 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,557 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:53:51,557 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:53:51,558 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:51,558 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:51,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:51,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:51,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:51,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:51,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:51,560 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:53:51,560 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4db8bb8e[Count = 0] remaining members to acquire global barrier 2023-04-12 15:53:51,560 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:53:51,560 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:51,562 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:51,562 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:53:51,563 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,563 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:53:51,563 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:51,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:51,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:51,563 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:53:51,563 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:53:51,563 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:53:51,566 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,566 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:53:51,566 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:53:51,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:51,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:51,566 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:53:51,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:51,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:51,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:51,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:51,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:51,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:53:51,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:53:51,569 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:53:51,569 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:53:51,569 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:53:51,569 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:53:51,569 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:53:51,571 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:51,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:51,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:51,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:51,571 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:51,571 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:53:51,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:51,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:51,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:51,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:53:51,572 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:53:51,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:51,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:51,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:51,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:51,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:51,579 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:51,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,584 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:53:51,584 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,584 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:53:51,584 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:51,584 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:51,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:53:51,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:53:51,584 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,584 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:53:51,584 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:53:51,584 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:53:51,584 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:51,585 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:51,585 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:53:51,585 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:53:51,585 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:51,588 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:51,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:53:51,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:53:51,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58340 deadline: 1681314891587, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,594 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:51,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58414 deadline: 1681314891592, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,685 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:53:51,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:53:51,688 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:53:51,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:53:51,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:53:51,689 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:53:51,690 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:53:51,690 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:53:51,690 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:53:51,690 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:51,690 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:51,692 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:53:51,692 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:53:51,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:53:51,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:51,693 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,693 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:53:51,693 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:51,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:53:51,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:51,694 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:53:51,696 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:53:51,696 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:53:51,696 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:53:51,696 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:53:51,696 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:53:51,696 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:53:51,696 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:53:51,696 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:51,697 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. started... 2023-04-12 15:53:51,697 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. as already flushing 2023-04-12 15:53:51,697 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:51,697 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:53:51,697 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:53:51,697 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,697 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:53:51,697 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:53:51,699 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:51,699 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:51,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:51,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:51,700 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:51,700 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:53:51,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:51,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:51,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:51,701 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:53:51,701 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3203f3ab[Count = 0] remaining members to acquire global barrier 2023-04-12 15:53:51,701 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:53:51,701 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:51,702 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:51,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:51,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:51,703 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:53:51,703 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:53:51,703 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:53:51,703 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,703 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:53:51,705 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:51,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:51,705 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:53:51,705 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:53:51,705 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:53:51,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:51,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:51,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:51,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:51,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:51,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:53:51,709 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:53:51,709 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:53:51,709 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:53:51,709 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:53:51,709 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:53:51,709 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:53:51,713 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:51,713 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:51,713 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:53:51,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:51,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:51,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:51,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:51,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:51,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:53:51,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:53:51,714 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:51,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:51,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:51,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:51,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:51,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,716 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:51,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:51,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,719 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:53:51,719 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:53:51,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:53:51,719 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:53:51,720 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:51,720 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:51,720 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,720 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:51,720 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:51,720 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:51,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:53:51,720 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:53:51,720 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:53:51,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:53:51,720 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:53:51,720 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:53:51,720 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:53:51,721 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:53:51,821 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:53:51,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:53:51,824 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:53:51,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:53:51,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:53:51,825 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:53:51,825 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:53:51,825 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:53:51,825 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:53:51,826 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:51,826 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:51,827 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,828 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:53:51,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:53:51,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:53:51,828 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,828 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:53:51,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:51,829 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:51,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:53:51,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:51,829 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:53:51,829 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:53:51,829 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:53:51,829 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:53:51,829 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:53:51,829 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:53:51,830 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:53:51,830 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:53:51,830 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:51,830 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. started... 2023-04-12 15:53:51,830 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. as already flushing 2023-04-12 15:53:51,830 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:51,830 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:53:51,830 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:53:51,830 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,830 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:53:51,830 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:53:51,831 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,832 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:51,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:51,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:51,832 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:51,832 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:53:51,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:51,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:51,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:51,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:51,836 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:53:51,836 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6da8a137[Count = 0] remaining members to acquire global barrier 2023-04-12 15:53:51,836 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:53:51,837 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:51,842 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:51,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:51,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:51,843 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:53:51,843 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,843 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:53:51,843 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:53:51,843 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:53:51,844 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,844 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:53:51,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:51,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:51,844 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:53:51,845 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:53:51,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:51,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:51,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:51,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:51,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:51,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:53:51,847 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:53:51,847 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:53:51,847 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:53:51,847 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:53:51,847 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:53:51,847 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:53:51,849 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:51,849 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:51,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:51,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:51,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:51,849 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:53:51,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:51,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:51,850 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:51,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:53:51,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:53:51,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:51,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:51,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:51,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:51,851 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:51,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:51,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,854 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:53:51,854 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:53:51,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:53:51,854 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:53:51,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:53:51,854 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,854 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:53:51,854 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:53:51,854 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:53:51,854 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:51,854 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:51,854 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,854 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:51,855 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:53:51,855 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:51,855 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:51,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:53:51,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:53:51,855 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:53:51,889 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=456 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/C/cc127c4e007d44e3875a39869cef4af6 2023-04-12 15:53:51,897 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/A/c2792699c7db48388370f7b2ee7057ef as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/c2792699c7db48388370f7b2ee7057ef 2023-04-12 15:53:51,904 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/c2792699c7db48388370f7b2ee7057ef, entries=150, sequenceid=456, filesize=11.9 K 2023-04-12 15:53:51,905 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/B/03499d929f184829aefcaafcd7ef27f5 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/03499d929f184829aefcaafcd7ef27f5 2023-04-12 15:53:51,912 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/03499d929f184829aefcaafcd7ef27f5, entries=150, sequenceid=456, filesize=11.9 K 2023-04-12 15:53:51,913 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/C/cc127c4e007d44e3875a39869cef4af6 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/cc127c4e007d44e3875a39869cef4af6 2023-04-12 15:53:51,919 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/cc127c4e007d44e3875a39869cef4af6, entries=150, sequenceid=456, filesize=11.9 K 2023-04-12 15:53:51,923 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~275.11 KB/281712, currentSize=80.51 KB/82440 for feda8274e3aaad09f0f3a1336e663d20 in 999ms, sequenceid=456, compaction requested=false 2023-04-12 15:53:51,923 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for feda8274e3aaad09f0f3a1336e663d20: 2023-04-12 15:53:51,955 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:53:51,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:53:51,958 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:53:51,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:53:51,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:53:51,960 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:53:51,960 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:53:51,960 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:53:51,960 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:53:51,963 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:51,963 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:51,965 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:53:51,965 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:53:51,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:53:51,966 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:51,966 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:53:51,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:51,966 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:51,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:53:51,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:51,967 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:53:51,967 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:53:51,967 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:53:51,967 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:53:51,967 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:53:51,967 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:53:51,968 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:53:51,968 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:53:51,968 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:51,968 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. started... 2023-04-12 15:53:51,968 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing feda8274e3aaad09f0f3a1336e663d20 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-12 15:53:51,968 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK feda8274e3aaad09f0f3a1336e663d20, store=A 2023-04-12 15:53:51,968 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:53:51,968 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK feda8274e3aaad09f0f3a1336e663d20, store=B 2023-04-12 15:53:51,968 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:53:51,968 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK feda8274e3aaad09f0f3a1336e663d20, store=C 2023-04-12 15:53:51,968 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:53:51,987 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=474 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/A/7588c177f7c04a76abfe8543746e3c3d 2023-04-12 15:53:52,012 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=474 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/B/0664ad13f11a4e5da35911f2db3f9a28 2023-04-12 15:53:52,028 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=474 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/C/1eb34d0ed2534b92a56256d196c0b691 2023-04-12 15:53:52,035 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/A/7588c177f7c04a76abfe8543746e3c3d as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/7588c177f7c04a76abfe8543746e3c3d 2023-04-12 15:53:52,049 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/7588c177f7c04a76abfe8543746e3c3d, entries=150, sequenceid=474, filesize=11.9 K 2023-04-12 15:53:52,057 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/B/0664ad13f11a4e5da35911f2db3f9a28 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/0664ad13f11a4e5da35911f2db3f9a28 2023-04-12 15:53:52,072 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/0664ad13f11a4e5da35911f2db3f9a28, entries=150, sequenceid=474, filesize=11.9 K 2023-04-12 15:53:52,074 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/C/1eb34d0ed2534b92a56256d196c0b691 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/1eb34d0ed2534b92a56256d196c0b691 2023-04-12 15:53:52,086 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/1eb34d0ed2534b92a56256d196c0b691, entries=150, sequenceid=474, filesize=11.9 K 2023-04-12 15:53:52,087 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=0 B/0 for feda8274e3aaad09f0f3a1336e663d20 in 119ms, sequenceid=474, compaction requested=true 2023-04-12 15:53:52,087 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for feda8274e3aaad09f0f3a1336e663d20: 2023-04-12 15:53:52,087 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:52,087 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:53:52,089 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:53:52,089 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,089 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:53:52,089 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:53:52,092 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,092 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:52,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:52,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:52,093 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:52,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:52,093 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:53:52,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:52,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:52,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:52,095 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:53:52,095 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76b8a12d[Count = 0] remaining members to acquire global barrier 2023-04-12 15:53:52,095 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:53:52,095 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:52,097 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:52,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:52,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:52,098 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:53:52,098 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:53:52,098 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:53:52,098 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,098 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:53:52,099 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:53:52,099 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,099 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:53:52,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:52,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:52,100 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:53:52,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:52,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:52,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:52,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:52,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:52,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:53:52,102 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:53:52,102 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:53:52,102 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:53:52,102 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:53:52,102 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:53:52,102 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:53:52,107 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:52,107 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:53:52,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:52,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:52,107 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:52,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:52,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:52,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:52,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:53:52,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:53:52,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:52,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:52,109 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:52,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:52,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:52,114 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:52,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:52,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,117 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:53:52,117 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:53:52,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:53:52,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:53:52,117 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,117 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:52,117 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:52,117 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,117 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:52,117 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:52,117 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:52,117 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:53:52,117 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:53:52,118 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:53:52,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:53:52,118 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:53:52,118 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:53:52,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:53:52,118 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:53:52,142 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing feda8274e3aaad09f0f3a1336e663d20 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-12 15:53:52,142 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK feda8274e3aaad09f0f3a1336e663d20, store=A 2023-04-12 15:53:52,142 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:53:52,142 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK feda8274e3aaad09f0f3a1336e663d20, store=B 2023-04-12 15:53:52,142 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:53:52,142 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK feda8274e3aaad09f0f3a1336e663d20, store=C 2023-04-12 15:53:52,142 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:53:52,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on feda8274e3aaad09f0f3a1336e663d20 2023-04-12 15:53:52,218 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:53:52,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:53:52,222 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681314832203, max timestamp=1681314832218Num uniques 859; 2023-04-12 15:53:52,223 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-12 15:53:52,223 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681314832203, max timestamp=1681314832218Num uniques 859; 2023-04-12 15:53:52,223 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-12 15:53:52,223 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681314832203, max timestamp=1681314832218Num uniques 859; 2023-04-12 15:53:52,223 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-12 15:53:52,224 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:53:52,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:53:52,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:53:52,240 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:53:52,240 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:53:52,240 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:53:52,240 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:53:52,241 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:52,241 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:52,243 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:53:52,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:53:52,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:53:52,243 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:52,244 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,244 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:53:52,244 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:52,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:53:52,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:52,245 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:53:52,246 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:53:52,246 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:53:52,246 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:53:52,246 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:53:52,248 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:53:52,248 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:53:52,248 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:52,248 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. started... 2023-04-12 15:53:52,248 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. as already flushing 2023-04-12 15:53:52,248 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:52,248 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:53:52,248 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:53:52,248 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,248 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:53:52,248 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:53:52,260 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,260 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:52,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:52,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:52,260 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:52,260 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:53:52,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:52,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:52,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:52,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:52,262 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:53:52,262 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ed400bf[Count = 0] remaining members to acquire global barrier 2023-04-12 15:53:52,262 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:53:52,262 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:52,263 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:52,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:52,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:52,264 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:53:52,264 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:53:52,264 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:53:52,264 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,264 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:53:52,265 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:53:52,265 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:53:52,265 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,265 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:53:52,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:52,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:52,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:52,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:52,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:52,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:52,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:52,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,268 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:52,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:53:52,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58340 deadline: 1681314892262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,268 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:53:52,268 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:53:52,268 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:53:52,268 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:53:52,268 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:53:52,268 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:53:52,270 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:52,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58414 deadline: 1681314892268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,271 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:52,271 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:53:52,271 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:52,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:52,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:52,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:52,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:52,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:52,272 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:53:52,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:52,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:53:52,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:52,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:52,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:52,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:52,273 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:52,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:52,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,277 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,277 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:52,277 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:53:52,277 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:53:52,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:53:52,277 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:52,277 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:53:52,277 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:53:52,277 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:53:52,277 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:53:52,277 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:52,277 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:52,277 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:52,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:53:52,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:53:52,278 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:53:52,278 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:53:52,370 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:52,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58340 deadline: 1681314892369, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,373 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:52,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58414 deadline: 1681314892372, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,378 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:53:52,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:53:52,381 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:53:52,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:53:52,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:53:52,382 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:53:52,382 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:53:52,382 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:53:52,382 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:53:52,383 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:52,383 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:52,384 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:53:52,384 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:53:52,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:53:52,384 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,384 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:53:52,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:52,385 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:52,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:53:52,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:52,385 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:53:52,386 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:53:52,386 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:53:52,386 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:53:52,386 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:53:52,386 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:53:52,386 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:53:52,386 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:53:52,386 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:52,386 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. started... 2023-04-12 15:53:52,386 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. as already flushing 2023-04-12 15:53:52,386 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:52,386 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:53:52,386 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:53:52,386 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,386 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:53:52,386 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:53:52,388 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:52,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:52,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:52,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:52,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:52,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:52,390 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:53:52,390 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53e73a65[Count = 0] remaining members to acquire global barrier 2023-04-12 15:53:52,390 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:53:52,390 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:52,391 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,391 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:53:52,396 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:52,396 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:52,396 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:52,396 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:53:52,396 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:53:52,396 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:53:52,396 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:53:52,400 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,400 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:53:52,400 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:53:52,400 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:53:52,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:52,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:52,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:52,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:52,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:52,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:52,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:52,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:53:52,405 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:53:52,405 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:53:52,405 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:53:52,405 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:53:52,405 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:53:52,405 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:53:52,408 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:52,408 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:52,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:52,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:52,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:52,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:52,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:52,408 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:53:52,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:52,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:53:52,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:53:52,409 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:52,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:52,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:52,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:52,412 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:52,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:52,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,415 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,415 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:52,415 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:52,415 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,415 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:53:52,415 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:53:52,415 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:53:52,415 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:53:52,415 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:53:52,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:53:52,416 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:53:52,416 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:53:52,415 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:52,415 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:52,416 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:53:52,416 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:52,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:53:52,416 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:52,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:53:52,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:53:52,460 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:52,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58374 deadline: 1681314892457, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,462 DEBUG [Thread-143] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4178 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20., hostname=jenkins-hbase4.apache.org,38905,1681314818739, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-12 15:53:52,464 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:52,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58398 deadline: 1681314892462, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,466 DEBUG [Thread-141] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4181 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20., hostname=jenkins-hbase4.apache.org,38905,1681314818739, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-12 15:53:52,480 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:52,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58354 deadline: 1681314892480, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,482 DEBUG [Thread-149] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4196 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20., hostname=jenkins-hbase4.apache.org,38905,1681314818739, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-12 15:53:52,516 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:53:52,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:53:52,520 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:53:52,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:53:52,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:53:52,521 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:53:52,521 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:53:52,522 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:53:52,522 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:53:52,522 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:52,522 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:52,524 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:53:52,524 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:53:52,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:53:52,524 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,525 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:53:52,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:52,525 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:52,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:53:52,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:52,526 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:53:52,526 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:53:52,526 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:53:52,526 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:53:52,526 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:53:52,526 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:53:52,526 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:53:52,526 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:53:52,526 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:52,527 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. started... 2023-04-12 15:53:52,527 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. as already flushing 2023-04-12 15:53:52,527 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:52,527 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:53:52,527 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:53:52,527 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,527 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:53:52,527 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:53:52,530 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,530 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:52,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:52,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:52,531 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:52,531 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:53:52,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:52,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:52,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:52,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:52,533 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:53:52,533 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7bee9837[Count = 0] remaining members to acquire global barrier 2023-04-12 15:53:52,533 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:53:52,533 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:52,534 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:52,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:52,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:52,535 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:53:52,535 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:53:52,535 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,535 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:53:52,535 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:53:52,537 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:53:52,537 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:53:52,537 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,537 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:53:52,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:52,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:52,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:52,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:52,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:52,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:52,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:52,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:53:52,544 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:53:52,544 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:53:52,544 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:53:52,544 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:53:52,544 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:53:52,544 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:53:52,545 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:52,545 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:53:52,546 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:52,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:52,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:52,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:52,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:52,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:52,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:52,547 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:53:52,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:53:52,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:52,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:52,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:52,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:52,549 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:52,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:52,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,556 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,557 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:52,557 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:52,557 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,557 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:53:52,557 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:53:52,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:53:52,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:53:52,557 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:53:52,557 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:53:52,557 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:53:52,557 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:53:52,558 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:53:52,557 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:52,558 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:52,558 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:52,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:53:52,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:53:52,576 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:52,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58340 deadline: 1681314892573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,577 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:52,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58414 deadline: 1681314892575, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,636 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=484 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/A/95667d8013f0420db185fb72810d9fe8 2023-04-12 15:53:52,658 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:53:52,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:53:52,660 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=484 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/B/018e9d42612741f0ae1376b89b682c02 2023-04-12 15:53:52,662 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:53:52,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:53:52,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:53:52,664 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:53:52,664 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:53:52,664 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:53:52,664 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:53:52,665 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:52,665 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:52,666 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,667 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:53:52,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:53:52,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:53:52,668 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:52,668 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:53:52,668 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:52,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:53:52,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:52,669 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:53:52,669 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:53:52,669 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:53:52,669 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:53:52,669 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:53:52,669 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:53:52,669 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:53:52,669 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:53:52,670 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:52,670 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. started... 2023-04-12 15:53:52,670 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. as already flushing 2023-04-12 15:53:52,670 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:52,670 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:53:52,670 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:53:52,670 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,670 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:53:52,670 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:53:52,674 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,674 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:52,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:52,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:52,675 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:52,675 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:53:52,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:52,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:52,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:52,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:52,676 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:53:52,676 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:53:52,676 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48ecefae[Count = 0] remaining members to acquire global barrier 2023-04-12 15:53:52,676 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:52,685 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:52,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:52,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:52,685 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,685 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:53:52,685 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:53:52,685 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:53:52,685 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:53:52,687 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:53:52,687 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,687 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:53:52,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:52,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:52,688 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:53:52,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:52,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:52,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:52,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:52,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:52,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:53:52,691 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:53:52,691 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:53:52,691 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:53:52,691 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:53:52,691 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:53:52,691 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:53:52,693 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:52,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:52,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:52,693 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:52,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:52,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:52,693 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:53:52,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:52,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:52,693 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:53:52,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:53:52,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:52,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:52,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:52,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:52,695 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:52,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:52,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,699 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,699 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:53:52,699 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:52,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:53:52,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:53:52,699 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:53:52,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:53:52,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:53:52,699 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:53:52,699 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:53:52,699 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:52,699 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:53:52,699 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,700 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:52,700 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:52,700 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:52,700 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:53:52,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:53:52,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:53:52,799 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:53:52,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:53:52,802 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:53:52,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:53:52,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:53:52,804 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:53:52,804 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:53:52,804 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:53:52,804 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:53:52,804 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:52,804 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:52,806 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:53:52,806 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:53:52,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:53:52,808 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,808 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:53:52,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:52,808 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:52,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:53:52,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:52,809 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:53:52,809 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:53:52,809 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:53:52,809 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:53:52,809 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:53:52,809 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:53:52,809 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:53:52,809 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:53:52,809 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:52,810 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. started... 2023-04-12 15:53:52,810 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. as already flushing 2023-04-12 15:53:52,810 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:52,810 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:53:52,810 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:53:52,810 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,810 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:53:52,810 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:53:52,811 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,811 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:52,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:52,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:52,812 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:52,812 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:53:52,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:52,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:52,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:52,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:52,813 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:53:52,814 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16e3dff4[Count = 0] remaining members to acquire global barrier 2023-04-12 15:53:52,814 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:53:52,814 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:52,815 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:52,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:52,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:52,815 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:53:52,815 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:53:52,815 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:53:52,815 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,815 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:53:52,817 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,817 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:53:52,817 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:53:52,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:52,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:52,817 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:53:52,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:52,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:52,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:52,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:52,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:52,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:53:52,825 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:53:52,825 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:53:52,825 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:53:52,825 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:53:52,825 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:53:52,825 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:53:52,827 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:52,827 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:52,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:52,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:52,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:52,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:52,827 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:53:52,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:52,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:52,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:53:52,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:53:52,828 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:52,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:52,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:52,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:52,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,830 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:52,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:52,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,833 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:53:52,833 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:53:52,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:53:52,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:53:52,833 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:53:52,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:53:52,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:53:52,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:53:52,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:53:52,833 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:53:52,833 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:53:52,836 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,836 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:52,836 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:52,836 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,836 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:52,836 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:52,836 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:52,882 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:52,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58340 deadline: 1681314892878, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,882 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:52,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58414 deadline: 1681314892880, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,933 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:53:52,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:53:52,936 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:53:52,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:53:52,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:53:52,938 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:53:52,938 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:53:52,938 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:53:52,938 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:53:52,938 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:52,938 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:52,939 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,939 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:53:52,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:53:52,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:53:52,940 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,940 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:53:52,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:52,940 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:52,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:53:52,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:52,941 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:53:52,941 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:53:52,941 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:53:52,941 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:53:52,941 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:53:52,941 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:53:52,941 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:53:52,941 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:53:52,941 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:52,941 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. started... 2023-04-12 15:53:52,941 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. as already flushing 2023-04-12 15:53:52,941 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:52,941 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:53:52,942 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:53:52,942 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,942 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:53:52,942 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:53:52,944 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,944 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:52,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:52,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:52,944 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:52,944 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:53:52,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:52,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:52,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:52,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:52,946 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:53:52,946 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:53:52,946 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c3dfb5f[Count = 0] remaining members to acquire global barrier 2023-04-12 15:53:52,946 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:52,947 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:52,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:52,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:52,947 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:53:52,947 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,948 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:53:52,947 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:53:52,948 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:53:52,949 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,949 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:53:52,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:52,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:52,949 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:53:52,949 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:53:52,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:52,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:52,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:52,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:52,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:52,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:53:52,952 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:53:52,952 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:53:52,952 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:53:52,952 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:53:52,952 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:53:52,952 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:53:52,954 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:52,954 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:52,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:52,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:52,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:52,954 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:53:52,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:52,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:52,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:52,954 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:53:52,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:53:52,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:52,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:52,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:52,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:52,956 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:52,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:52,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,959 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,959 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:53:52,959 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:52,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:53:52,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:53:52,959 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:52,959 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:52,959 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:52,959 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:52,959 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:52,959 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:53:52,959 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:53:52,959 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:53:52,959 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:53:52,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:53:52,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:53:52,959 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:53:52,959 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:53:52,959 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:53:53,060 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:53:53,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:53:53,062 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:53:53,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:53:53,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:53:53,064 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:53:53,064 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:53:53,064 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:53:53,064 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:53:53,064 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:53,064 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:53,066 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:53:53,066 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:53,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:53:53,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:53:53,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:53,066 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:53,066 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:53:53,067 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:53,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:53:53,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:53,067 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:53:53,067 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:53:53,067 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:53:53,067 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:53:53,068 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:53:53,068 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:53:53,068 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:53:53,068 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:53:53,068 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:53,070 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. started... 2023-04-12 15:53:53,070 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. as already flushing 2023-04-12 15:53:53,070 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:53,070 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:53:53,070 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:53:53,070 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:53,070 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:53:53,070 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:53:53,079 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:53,079 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:53,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:53,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:53,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:53,079 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:53,079 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:53:53,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:53,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:53,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:53,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:53,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:53,082 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:53:53,082 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@731debcf[Count = 0] remaining members to acquire global barrier 2023-04-12 15:53:53,082 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:53:53,082 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:53,083 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=484 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/C/d09fb7e12b214e8e83c988672afa3a54 2023-04-12 15:53:53,084 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:53,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:53,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:53,084 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:53:53,084 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:53:53,084 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:53:53,085 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:53,085 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:53:53,089 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:53,089 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:53:53,089 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:53:53,089 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:53:53,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:53,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:53,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:53,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:53,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:53,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:53,091 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/A/95667d8013f0420db185fb72810d9fe8 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/95667d8013f0420db185fb72810d9fe8 2023-04-12 15:53:53,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:53,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:53,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:53,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:53,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:53:53,094 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:53:53,094 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:53:53,094 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:53:53,094 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:53:53,094 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:53:53,094 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:53:53,096 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:53,096 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:53,096 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:53:53,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:53,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:53,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:53,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:53,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:53,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:53:53,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:53:53,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:53,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:53,098 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:53,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:53,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:53,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:53,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:53,100 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:53,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:53,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:53,100 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/95667d8013f0420db185fb72810d9fe8, entries=200, sequenceid=484, filesize=14.3 K 2023-04-12 15:53:53,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:53,102 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/B/018e9d42612741f0ae1376b89b682c02 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/018e9d42612741f0ae1376b89b682c02 2023-04-12 15:53:53,103 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:53,103 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:53,103 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:53,103 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:53,103 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:53,103 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:53,103 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:53,103 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:53:53,103 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:53:53,103 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:53:53,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:53:53,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:53:53,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:53:53,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:53:53,104 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:53:53,104 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:53:53,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:53:53,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:53:53,108 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/018e9d42612741f0ae1376b89b682c02, entries=100, sequenceid=484, filesize=9.5 K 2023-04-12 15:53:53,110 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/C/d09fb7e12b214e8e83c988672afa3a54 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/d09fb7e12b214e8e83c988672afa3a54 2023-04-12 15:53:53,116 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/d09fb7e12b214e8e83c988672afa3a54, entries=100, sequenceid=484, filesize=9.5 K 2023-04-12 15:53:53,118 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for feda8274e3aaad09f0f3a1336e663d20 in 976ms, sequenceid=484, compaction requested=true 2023-04-12 15:53:53,118 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for feda8274e3aaad09f0f3a1336e663d20: 2023-04-12 15:53:53,119 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:53:53,119 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-12 15:53:53,119 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-12 15:53:53,119 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:53:53,119 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:53:53,120 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 51522 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-12 15:53:53,121 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1912): feda8274e3aaad09f0f3a1336e663d20/A is initiating minor compaction (all files) 2023-04-12 15:53:53,121 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of feda8274e3aaad09f0f3a1336e663d20/A in TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:53,121 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/96e5ebbbdecd47f4a574785d8f4e4ea2, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/c2792699c7db48388370f7b2ee7057ef, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/7588c177f7c04a76abfe8543746e3c3d, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/95667d8013f0420db185fb72810d9fe8] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp, totalSize=50.3 K 2023-04-12 15:53:53,121 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46642 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-12 15:53:53,121 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1912): feda8274e3aaad09f0f3a1336e663d20/B is initiating minor compaction (all files) 2023-04-12 15:53:53,121 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2259): Starting compaction of feda8274e3aaad09f0f3a1336e663d20/B in TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:53,121 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/fe8d0f5e333c407d9cbd1335556a698b, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/03499d929f184829aefcaafcd7ef27f5, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/0664ad13f11a4e5da35911f2db3f9a28, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/018e9d42612741f0ae1376b89b682c02] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp, totalSize=45.5 K 2023-04-12 15:53:53,122 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 96e5ebbbdecd47f4a574785d8f4e4ea2, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=409, earliestPutTs=1681314830435 2023-04-12 15:53:53,124 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting fe8d0f5e333c407d9cbd1335556a698b, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=409, earliestPutTs=1681314830435 2023-04-12 15:53:53,125 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting c2792699c7db48388370f7b2ee7057ef, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=456, earliestPutTs=1681314830902 2023-04-12 15:53:53,126 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 03499d929f184829aefcaafcd7ef27f5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=456, earliestPutTs=1681314830902 2023-04-12 15:53:53,126 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 7588c177f7c04a76abfe8543746e3c3d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=474, earliestPutTs=1681314830953 2023-04-12 15:53:53,126 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 0664ad13f11a4e5da35911f2db3f9a28, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=474, earliestPutTs=1681314830953 2023-04-12 15:53:53,126 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 95667d8013f0420db185fb72810d9fe8, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=484, earliestPutTs=1681314832097 2023-04-12 15:53:53,127 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 018e9d42612741f0ae1376b89b682c02, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=484, earliestPutTs=1681314832105 2023-04-12 15:53:53,145 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] throttle.PressureAwareThroughputController(145): feda8274e3aaad09f0f3a1336e663d20#A#compaction#43 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:53:53,146 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] throttle.PressureAwareThroughputController(145): feda8274e3aaad09f0f3a1336e663d20#B#compaction#42 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:53:53,204 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:53:53,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:53:53,206 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:53:53,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:53:53,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:53:53,207 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:53:53,208 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:53:53,208 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:53:53,208 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:53:53,208 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:53,208 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:53,210 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:53:53,210 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:53,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:53:53,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:53:53,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:53,211 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:53,211 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:53:53,212 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:53,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:53:53,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:53,213 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:53:53,216 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:53:53,216 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:53:53,216 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:53:53,216 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:53:53,216 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:53:53,216 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:53:53,216 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:53:53,216 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:53,216 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. started... 2023-04-12 15:53:53,216 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing feda8274e3aaad09f0f3a1336e663d20 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-12 15:53:53,217 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK feda8274e3aaad09f0f3a1336e663d20, store=A 2023-04-12 15:53:53,217 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:53:53,217 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK feda8274e3aaad09f0f3a1336e663d20, store=B 2023-04-12 15:53:53,217 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:53:53,217 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK feda8274e3aaad09f0f3a1336e663d20, store=C 2023-04-12 15:53:53,217 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:53:53,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on feda8274e3aaad09f0f3a1336e663d20 2023-04-12 15:53:53,388 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. as already flushing 2023-04-12 15:53:53,404 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:53,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58340 deadline: 1681314893400, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:53,409 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:53,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58414 deadline: 1681314893404, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:53,508 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:53,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58340 deadline: 1681314893507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:53,511 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:53,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58414 deadline: 1681314893510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:53,567 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/B/44460f66245640c8835841e0145d0671 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/44460f66245640c8835841e0145d0671 2023-04-12 15:53:53,580 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/A/98652c09fc6f47709fbb1ffcd0800b8e as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/98652c09fc6f47709fbb1ffcd0800b8e 2023-04-12 15:53:53,586 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in feda8274e3aaad09f0f3a1336e663d20/B of feda8274e3aaad09f0f3a1336e663d20 into 44460f66245640c8835841e0145d0671(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:53:53,587 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for feda8274e3aaad09f0f3a1336e663d20: 2023-04-12 15:53:53,587 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20., storeName=feda8274e3aaad09f0f3a1336e663d20/B, priority=12, startTime=1681314833119; duration=0sec 2023-04-12 15:53:53,587 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:53:53,587 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-12 15:53:53,590 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46642 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-12 15:53:53,591 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1912): feda8274e3aaad09f0f3a1336e663d20/C is initiating minor compaction (all files) 2023-04-12 15:53:53,591 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2259): Starting compaction of feda8274e3aaad09f0f3a1336e663d20/C in TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:53,591 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/10feb66066be4a55bb8f1a2f2998be04, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/cc127c4e007d44e3875a39869cef4af6, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/1eb34d0ed2534b92a56256d196c0b691, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/d09fb7e12b214e8e83c988672afa3a54] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp, totalSize=45.5 K 2023-04-12 15:53:53,591 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 10feb66066be4a55bb8f1a2f2998be04, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=409, earliestPutTs=1681314830435 2023-04-12 15:53:53,591 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in feda8274e3aaad09f0f3a1336e663d20/A of feda8274e3aaad09f0f3a1336e663d20 into 98652c09fc6f47709fbb1ffcd0800b8e(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:53:53,593 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for feda8274e3aaad09f0f3a1336e663d20: 2023-04-12 15:53:53,593 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20., storeName=feda8274e3aaad09f0f3a1336e663d20/A, priority=12, startTime=1681314833119; duration=0sec 2023-04-12 15:53:53,593 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:53:53,594 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting cc127c4e007d44e3875a39869cef4af6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=456, earliestPutTs=1681314830902 2023-04-12 15:53:53,595 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 1eb34d0ed2534b92a56256d196c0b691, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=474, earliestPutTs=1681314830953 2023-04-12 15:53:53,596 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting d09fb7e12b214e8e83c988672afa3a54, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=484, earliestPutTs=1681314832105 2023-04-12 15:53:53,620 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] throttle.PressureAwareThroughputController(145): feda8274e3aaad09f0f3a1336e663d20#C#compaction#45 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:53:53,633 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=520 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/A/538b0f0b3a6d478ab26459ed27598c7b 2023-04-12 15:53:53,672 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/C/dc35c07fb7994813b6248e58273954ed as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/dc35c07fb7994813b6248e58273954ed 2023-04-12 15:53:53,679 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=520 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/B/8be84eb2f3b1437a964861c66b5ee6ee 2023-04-12 15:53:53,681 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in feda8274e3aaad09f0f3a1336e663d20/C of feda8274e3aaad09f0f3a1336e663d20 into dc35c07fb7994813b6248e58273954ed(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:53:53,681 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for feda8274e3aaad09f0f3a1336e663d20: 2023-04-12 15:53:53,681 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20., storeName=feda8274e3aaad09f0f3a1336e663d20/C, priority=12, startTime=1681314833119; duration=0sec 2023-04-12 15:53:53,681 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:53:53,714 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:53,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58340 deadline: 1681314893711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:53,714 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:53,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58414 deadline: 1681314893713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:53,733 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=520 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/C/b985bf1b49d94215a9efe19d9a8e1895 2023-04-12 15:53:53,742 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/A/538b0f0b3a6d478ab26459ed27598c7b as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/538b0f0b3a6d478ab26459ed27598c7b 2023-04-12 15:53:53,750 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/538b0f0b3a6d478ab26459ed27598c7b, entries=150, sequenceid=520, filesize=11.9 K 2023-04-12 15:53:53,752 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/B/8be84eb2f3b1437a964861c66b5ee6ee as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/8be84eb2f3b1437a964861c66b5ee6ee 2023-04-12 15:53:53,758 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/8be84eb2f3b1437a964861c66b5ee6ee, entries=150, sequenceid=520, filesize=11.9 K 2023-04-12 15:53:53,759 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/C/b985bf1b49d94215a9efe19d9a8e1895 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/b985bf1b49d94215a9efe19d9a8e1895 2023-04-12 15:53:53,771 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/b985bf1b49d94215a9efe19d9a8e1895, entries=150, sequenceid=520, filesize=11.9 K 2023-04-12 15:53:53,772 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for feda8274e3aaad09f0f3a1336e663d20 in 556ms, sequenceid=520, compaction requested=false 2023-04-12 15:53:53,772 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for feda8274e3aaad09f0f3a1336e663d20: 2023-04-12 15:53:53,773 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:53,773 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:53:53,778 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:53:53,778 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:53,778 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:53:53,778 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:53:53,780 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:53,780 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:53,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:53,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:53,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:53,781 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:53,781 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:53:53,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:53,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:53,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:53,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:53,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:53,783 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:53:53,783 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c23fb9d[Count = 0] remaining members to acquire global barrier 2023-04-12 15:53:53,783 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:53:53,783 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:53,784 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:53,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:53,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:53,785 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:53:53,785 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:53:53,785 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:53:53,785 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:53,785 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:53:53,786 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:53,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:53,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:53,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:53,787 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:53:53,787 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:53:53,787 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:53:53,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:53,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:53,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:53,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:53,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:53,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:53,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:53,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:53:53,790 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:53:53,790 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:53:53,790 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:53:53,791 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:53:53,791 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:53:53,791 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:53:53,794 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:53,794 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:53,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:53,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:53,795 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:53:53,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:53,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:53,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:53,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:53:53,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:53:53,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:53,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:53,801 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:53,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:53,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:53,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:53,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:53,813 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:53,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:53,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:53,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:53,818 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:53:53,818 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:53:53,818 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:53:53,818 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:53:53,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:53:53,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:53:53,818 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:53:53,819 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:53:53,819 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:53:53,818 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:53,820 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:53,820 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:53,820 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:53,820 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:53,820 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:53,820 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:53,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:53:53,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:53:53,919 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:53:53,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:53:53,923 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:53:53,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:53:53,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:53:53,924 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:53:53,925 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:53:53,925 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:53:53,925 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:53:53,925 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:53,925 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:53,927 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:53:53,927 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:53,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:53:53,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:53:53,927 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:53,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:53,927 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:53:53,928 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:53,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:53:53,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:53,928 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:53:53,933 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:53:53,933 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:53:53,933 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:53:53,933 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:53:53,933 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:53:53,937 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:53:53,937 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:53:53,937 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:53,937 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. started... 2023-04-12 15:53:53,938 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing feda8274e3aaad09f0f3a1336e663d20 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-12 15:53:53,938 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK feda8274e3aaad09f0f3a1336e663d20, store=A 2023-04-12 15:53:53,938 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:53:53,938 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK feda8274e3aaad09f0f3a1336e663d20, store=B 2023-04-12 15:53:53,938 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:53:53,938 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK feda8274e3aaad09f0f3a1336e663d20, store=C 2023-04-12 15:53:53,938 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:53:53,951 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=533 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/A/31b3c6bb09a8434596a142a1de704dad 2023-04-12 15:53:53,975 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=533 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/B/6dca3d7853cf4752829affa84c094274 2023-04-12 15:53:54,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on feda8274e3aaad09f0f3a1336e663d20 2023-04-12 15:53:54,019 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. as already flushing 2023-04-12 15:53:54,057 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681314834046, max timestamp=1681314834055Num uniques 859; 2023-04-12 15:53:54,057 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681314834046, max timestamp=1681314834055Num uniques 859; 2023-04-12 15:53:54,057 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-12 15:53:54,057 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681314834046, max timestamp=1681314834055Num uniques 859; 2023-04-12 15:53:54,058 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-12 15:53:54,058 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-12 15:53:54,103 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:54,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58414 deadline: 1681314894099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:54,103 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:54,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58340 deadline: 1681314894101, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:54,210 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:54,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58414 deadline: 1681314894207, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:54,211 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:54,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58340 deadline: 1681314894210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:54,394 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=533 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/C/18a512a078e9487486d16cc19306fe24 2023-04-12 15:53:54,411 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/A/31b3c6bb09a8434596a142a1de704dad as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/31b3c6bb09a8434596a142a1de704dad 2023-04-12 15:53:54,415 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:54,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58340 deadline: 1681314894413, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:54,416 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:54,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58414 deadline: 1681314894414, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:54,420 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/31b3c6bb09a8434596a142a1de704dad, entries=150, sequenceid=533, filesize=11.9 K 2023-04-12 15:53:54,421 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/B/6dca3d7853cf4752829affa84c094274 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/6dca3d7853cf4752829affa84c094274 2023-04-12 15:53:54,430 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/6dca3d7853cf4752829affa84c094274, entries=150, sequenceid=533, filesize=11.9 K 2023-04-12 15:53:54,431 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/C/18a512a078e9487486d16cc19306fe24 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/18a512a078e9487486d16cc19306fe24 2023-04-12 15:53:54,438 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/18a512a078e9487486d16cc19306fe24, entries=150, sequenceid=533, filesize=11.9 K 2023-04-12 15:53:54,439 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for feda8274e3aaad09f0f3a1336e663d20 in 502ms, sequenceid=533, compaction requested=true 2023-04-12 15:53:54,439 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for feda8274e3aaad09f0f3a1336e663d20: 2023-04-12 15:53:54,439 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:54,439 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:53:54,439 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:53:54,439 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:54,439 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:53:54,439 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:53:54,445 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:54,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:54,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:54,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:54,445 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:54,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:54,445 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:54,445 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:53:54,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:54,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:54,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:54,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:54,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:53:54,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54379392[Count = 0] remaining members to acquire global barrier 2023-04-12 15:53:54,448 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:53:54,448 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:54,451 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:54,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:54,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:54,451 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:53:54,451 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:53:54,451 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:53:54,451 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:54,452 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:53:54,453 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:54,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:54,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:54,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:54,456 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:53:54,456 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:53:54,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:54,456 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:53:54,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:54,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:54,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:54,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:54,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:54,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:54,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:53:54,460 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:53:54,460 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:53:54,460 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:53:54,460 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:53:54,460 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:53:54,460 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:53:54,462 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:54,462 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:53:54,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:54,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:54,462 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:54,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:54,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:54,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:54,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:53:54,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:53:54,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:54,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:54,463 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:54,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:54,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:54,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:54,465 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:54,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:54,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:54,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:54,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:54,468 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:53:54,468 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:54,468 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:53:54,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:53:54,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:53:54,468 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:53:54,468 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:53:54,468 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:53:54,468 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:54,469 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:54,469 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:54,469 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:54,469 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:54,469 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:54,469 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:53:54,469 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:53:54,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:53:54,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:53:54,470 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:53:54,569 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:53:54,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:53:54,572 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:53:54,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:53:54,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:53:54,576 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:53:54,577 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:53:54,577 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:53:54,577 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:53:54,577 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:54,577 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:54,580 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:53:54,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:53:54,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:53:54,580 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:54,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:54,580 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:54,580 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:53:54,581 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:54,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:53:54,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:54,581 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:53:54,597 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:53:54,597 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:53:54,598 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:53:54,598 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:53:54,598 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:53:54,603 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:53:54,603 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:53:54,603 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:54,603 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. started... 2023-04-12 15:53:54,604 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing feda8274e3aaad09f0f3a1336e663d20 3/3 column families, dataSize=134.18 KB heapSize=375.61 KB 2023-04-12 15:53:54,604 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK feda8274e3aaad09f0f3a1336e663d20, store=A 2023-04-12 15:53:54,604 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:53:54,604 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK feda8274e3aaad09f0f3a1336e663d20, store=B 2023-04-12 15:53:54,604 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:53:54,604 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK feda8274e3aaad09f0f3a1336e663d20, store=C 2023-04-12 15:53:54,604 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:53:54,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on feda8274e3aaad09f0f3a1336e663d20 2023-04-12 15:53:54,719 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. as already flushing 2023-04-12 15:53:54,738 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:54,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58414 deadline: 1681314894734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:54,745 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:54,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58340 deadline: 1681314894743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:54,842 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:54,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58414 deadline: 1681314894840, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:54,854 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:54,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58340 deadline: 1681314894852, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:55,022 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=568 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/A/62693f50f24345fe8e5c1672db71a03b 2023-04-12 15:53:55,046 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:55,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58414 deadline: 1681314895044, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:55,060 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:55,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58340 deadline: 1681314895056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:55,354 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:55,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58414 deadline: 1681314895348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:55,365 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:55,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58340 deadline: 1681314895362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:55,474 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=568 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/B/6987831befee4dd7a95b462a1caab04c 2023-04-12 15:53:55,856 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:55,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58414 deadline: 1681314895855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:55,869 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:55,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58340 deadline: 1681314895868, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:55,889 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=568 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/C/b297403b25a042c39756052596873237 2023-04-12 15:53:55,897 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/A/62693f50f24345fe8e5c1672db71a03b as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/62693f50f24345fe8e5c1672db71a03b 2023-04-12 15:53:55,911 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/62693f50f24345fe8e5c1672db71a03b, entries=150, sequenceid=568, filesize=11.9 K 2023-04-12 15:53:55,913 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/B/6987831befee4dd7a95b462a1caab04c as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/6987831befee4dd7a95b462a1caab04c 2023-04-12 15:53:55,920 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/6987831befee4dd7a95b462a1caab04c, entries=150, sequenceid=568, filesize=11.9 K 2023-04-12 15:53:55,921 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/C/b297403b25a042c39756052596873237 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/b297403b25a042c39756052596873237 2023-04-12 15:53:55,927 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/b297403b25a042c39756052596873237, entries=150, sequenceid=568, filesize=11.9 K 2023-04-12 15:53:55,928 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~375.56 KB/384576, currentSize=46.96 KB/48090 for feda8274e3aaad09f0f3a1336e663d20 in 1324ms, sequenceid=568, compaction requested=true 2023-04-12 15:53:55,929 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for feda8274e3aaad09f0f3a1336e663d20: 2023-04-12 15:53:55,929 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:55,929 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:53:55,929 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:53:55,929 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:55,929 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:53:55,929 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:53:55,931 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:55,931 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:55,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:55,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:55,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:55,931 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:55,931 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:53:55,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:55,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:55,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:55,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:55,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:55,933 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:53:55,933 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47c37d48[Count = 0] remaining members to acquire global barrier 2023-04-12 15:53:55,933 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:53:55,933 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:55,935 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:55,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:55,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:55,935 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:53:55,935 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:53:55,935 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:53:55,936 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:55,936 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:53:55,938 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:55,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:55,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:55,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:55,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:55,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:55,939 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:53:55,939 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:53:55,939 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:53:55,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:55,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:55,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:55,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:55,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:55,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:53:55,948 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:53:55,948 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:53:55,948 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:53:55,948 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:53:55,948 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:53:55,948 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:53:55,951 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:55,951 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:53:55,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:55,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:55,951 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:55,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:55,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:55,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:55,952 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:55,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:53:55,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:53:55,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:55,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:55,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:55,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:55,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:55,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:55,955 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:55,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:55,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:55,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:55,961 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:53:55,961 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:53:55,961 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:55,961 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:53:55,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:53:55,961 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:55,961 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:53:55,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:53:55,961 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:53:55,962 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:55,962 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:53:55,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:53:55,962 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:53:55,962 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:55,962 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:55,962 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:55,962 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:55,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:53:55,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:53:56,062 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:53:56,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:53:56,065 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:53:56,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:53:56,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:53:56,070 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:53:56,070 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:53:56,070 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:53:56,070 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:53:56,071 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:56,071 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:56,073 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:53:56,073 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:56,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:53:56,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:53:56,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:56,074 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:56,074 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:53:56,075 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:56,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:53:56,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:56,076 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:53:56,076 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:53:56,076 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:53:56,076 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:53:56,076 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:53:56,076 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:53:56,078 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:53:56,078 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:53:56,078 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:56,078 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. started... 2023-04-12 15:53:56,079 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing feda8274e3aaad09f0f3a1336e663d20 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-12 15:53:56,079 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK feda8274e3aaad09f0f3a1336e663d20, store=A 2023-04-12 15:53:56,079 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:53:56,079 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK feda8274e3aaad09f0f3a1336e663d20, store=B 2023-04-12 15:53:56,079 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:53:56,079 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK feda8274e3aaad09f0f3a1336e663d20, store=C 2023-04-12 15:53:56,079 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:53:56,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on feda8274e3aaad09f0f3a1336e663d20 2023-04-12 15:53:56,475 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. as already flushing 2023-04-12 15:53:56,499 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=578 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/A/19892e6cb9534a64b07605bf9fa6deee 2023-04-12 15:53:56,516 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681314836488, max timestamp=1681314836513Num uniques 859; 2023-04-12 15:53:56,516 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-12 15:53:56,517 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681314836488, max timestamp=1681314836513Num uniques 859; 2023-04-12 15:53:56,517 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-12 15:53:56,517 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681314836488, max timestamp=1681314836513Num uniques 859; 2023-04-12 15:53:56,517 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-12 15:53:56,598 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:56,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58398 deadline: 1681314896591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:56,599 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:56,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58374 deadline: 1681314896593, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:56,600 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:56,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58354 deadline: 1681314896595, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:56,702 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:56,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58398 deadline: 1681314896700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:56,704 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:56,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58374 deadline: 1681314896701, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:56,707 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:56,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58354 deadline: 1681314896704, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:56,862 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:56,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58414 deadline: 1681314896861, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:56,879 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:56,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58340 deadline: 1681314896877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:56,906 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:56,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58398 deadline: 1681314896904, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:56,907 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:56,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58374 deadline: 1681314896906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:56,911 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:56,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58354 deadline: 1681314896908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:56,958 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=578 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/B/ce27e115c43f48aba2e29720eaec8653 2023-04-12 15:53:57,211 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:57,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58398 deadline: 1681314897209, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:57,212 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:57,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58374 deadline: 1681314897210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:57,215 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:57,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58354 deadline: 1681314897213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:57,379 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=578 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/C/d40b89f36464453ab1d40cce50cc40ec 2023-04-12 15:53:57,390 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/A/19892e6cb9534a64b07605bf9fa6deee as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/19892e6cb9534a64b07605bf9fa6deee 2023-04-12 15:53:57,399 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/19892e6cb9534a64b07605bf9fa6deee, entries=150, sequenceid=578, filesize=11.9 K 2023-04-12 15:53:57,402 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/B/ce27e115c43f48aba2e29720eaec8653 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/ce27e115c43f48aba2e29720eaec8653 2023-04-12 15:53:57,412 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/ce27e115c43f48aba2e29720eaec8653, entries=150, sequenceid=578, filesize=11.9 K 2023-04-12 15:53:57,413 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/C/d40b89f36464453ab1d40cce50cc40ec as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/d40b89f36464453ab1d40cce50cc40ec 2023-04-12 15:53:57,418 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/d40b89f36464453ab1d40cce50cc40ec, entries=150, sequenceid=578, filesize=11.9 K 2023-04-12 15:53:57,419 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for feda8274e3aaad09f0f3a1336e663d20 in 1339ms, sequenceid=578, compaction requested=true 2023-04-12 15:53:57,419 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for feda8274e3aaad09f0f3a1336e663d20: 2023-04-12 15:53:57,419 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:57,419 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:53:57,419 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:53:57,419 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:57,419 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:53:57,419 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:53:57,421 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:57,421 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:57,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:57,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:57,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:57,421 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:57,421 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:53:57,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:57,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:57,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:57,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:57,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:57,423 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:53:57,423 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b9ca5ec[Count = 0] remaining members to acquire global barrier 2023-04-12 15:53:57,423 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:53:57,423 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:57,425 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:57,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:57,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:57,425 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:53:57,425 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:53:57,425 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:53:57,425 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:57,425 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:53:57,427 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:53:57,427 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:57,427 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:53:57,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:57,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:57,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:57,428 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:53:57,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:57,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:57,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:57,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:57,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:57,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:57,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:57,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:53:57,430 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:53:57,430 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:53:57,430 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:53:57,430 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:53:57,430 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:53:57,430 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:53:57,432 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:57,432 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:57,432 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:53:57,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:57,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:57,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:57,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:57,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:57,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:57,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:53:57,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:53:57,433 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:57,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:57,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:57,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:57,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:57,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:57,434 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:57,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:57,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:57,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:57,436 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:57,436 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:57,436 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:57,436 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:53:57,436 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:53:57,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:53:57,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:53:57,436 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:57,436 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:57,436 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:57,436 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:53:57,436 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:53:57,436 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:57,436 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:53:57,436 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:53:57,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:53:57,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:53:57,437 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:53:57,437 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:53:57,537 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:53:57,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:53:57,540 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:53:57,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:53:57,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:53:57,566 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:53:57,566 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:53:57,566 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:53:57,566 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:53:57,566 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:57,566 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:57,568 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:53:57,568 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:57,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:53:57,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:53:57,569 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:57,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:57,569 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:53:57,569 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:57,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:53:57,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:57,570 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:53:57,572 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:53:57,572 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:53:57,572 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:53:57,572 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:53:57,572 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:53:57,578 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:53:57,578 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:53:57,578 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:57,578 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. started... 2023-04-12 15:53:57,578 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing feda8274e3aaad09f0f3a1336e663d20 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-12 15:53:57,578 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK feda8274e3aaad09f0f3a1336e663d20, store=A 2023-04-12 15:53:57,578 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:53:57,579 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK feda8274e3aaad09f0f3a1336e663d20, store=B 2023-04-12 15:53:57,579 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:53:57,579 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK feda8274e3aaad09f0f3a1336e663d20, store=C 2023-04-12 15:53:57,579 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:53:57,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on feda8274e3aaad09f0f3a1336e663d20 2023-04-12 15:53:57,719 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. as already flushing 2023-04-12 15:53:57,755 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:57,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58354 deadline: 1681314897752, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:57,755 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:57,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58398 deadline: 1681314897753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:57,760 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:57,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58374 deadline: 1681314897755, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:57,859 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:57,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58354 deadline: 1681314897856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:57,859 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:57,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58398 deadline: 1681314897856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:57,865 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:57,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58374 deadline: 1681314897861, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:57,993 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=614 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/A/3b54c347241c4d0eb27e2ab352d1fb72 2023-04-12 15:53:58,062 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:58,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58354 deadline: 1681314898060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:58,064 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:58,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58398 deadline: 1681314898062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:58,068 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:58,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58374 deadline: 1681314898066, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:58,370 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:58,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58354 deadline: 1681314898365, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:58,371 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:58,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58398 deadline: 1681314898365, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:58,371 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:58,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58374 deadline: 1681314898370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:58,410 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=614 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/B/3ad20f89347e488bb98f491671452d33 2023-04-12 15:53:58,826 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=614 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/C/5e234b651bf54320b2e8581163dbdcef 2023-04-12 15:53:58,833 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/A/3b54c347241c4d0eb27e2ab352d1fb72 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/3b54c347241c4d0eb27e2ab352d1fb72 2023-04-12 15:53:58,841 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/3b54c347241c4d0eb27e2ab352d1fb72, entries=150, sequenceid=614, filesize=11.9 K 2023-04-12 15:53:58,842 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/B/3ad20f89347e488bb98f491671452d33 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/3ad20f89347e488bb98f491671452d33 2023-04-12 15:53:58,850 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/3ad20f89347e488bb98f491671452d33, entries=150, sequenceid=614, filesize=11.9 K 2023-04-12 15:53:58,851 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/C/5e234b651bf54320b2e8581163dbdcef as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/5e234b651bf54320b2e8581163dbdcef 2023-04-12 15:53:58,857 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/5e234b651bf54320b2e8581163dbdcef, entries=150, sequenceid=614, filesize=11.9 K 2023-04-12 15:53:58,858 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for feda8274e3aaad09f0f3a1336e663d20 in 1280ms, sequenceid=614, compaction requested=true 2023-04-12 15:53:58,858 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for feda8274e3aaad09f0f3a1336e663d20: 2023-04-12 15:53:58,858 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:58,858 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:53:58,858 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:53:58,858 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:58,858 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:53:58,858 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:53:58,860 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:58,860 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:58,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:58,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:58,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:58,860 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:58,860 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:53:58,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:58,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:58,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:58,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:58,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:58,864 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:53:58,864 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18204211[Count = 0] remaining members to acquire global barrier 2023-04-12 15:53:58,864 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:53:58,864 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:58,865 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:58,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:58,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:58,865 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:53:58,865 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:53:58,865 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:53:58,866 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:58,866 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:53:58,867 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:53:58,867 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:58,867 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:53:58,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:58,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:58,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:58,867 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:53:58,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:58,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:58,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:58,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:58,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:58,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:58,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:58,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:53:58,870 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:53:58,870 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:53:58,870 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:53:58,870 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:53:58,870 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:53:58,870 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:53:58,874 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:58,874 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:53:58,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:58,874 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:58,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:58,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:53:58,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:53:58,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on feda8274e3aaad09f0f3a1336e663d20 2023-04-12 15:53:58,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:58,875 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing feda8274e3aaad09f0f3a1336e663d20 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-12 15:53:58,876 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:58,876 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK feda8274e3aaad09f0f3a1336e663d20, store=A 2023-04-12 15:53:58,877 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:53:58,877 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK feda8274e3aaad09f0f3a1336e663d20, store=B 2023-04-12 15:53:58,877 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:53:58,877 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK feda8274e3aaad09f0f3a1336e663d20, store=C 2023-04-12 15:53:58,877 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:53:58,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:58,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:58,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:58,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:58,880 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:58,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:58,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:58,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:58,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:58,883 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:53:58,883 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:53:58,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:53:58,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:53:58,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:58,884 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:53:58,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:53:58,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:53:58,884 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:53:58,884 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:53:58,884 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:58,884 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:58,884 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:58,884 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:58,884 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:58,884 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:58,884 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:58,884 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:53:58,885 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:53:58,885 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:53:58,948 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:58,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58354 deadline: 1681314898945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:58,948 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9554, totalHeapSize=21976, min timestamp=1681314838912, max timestamp=1681314838946Num uniques 859; 2023-04-12 15:53:58,948 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39324, new segment data size=9554, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-12 15:53:58,948 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9561, totalHeapSize=21976, min timestamp=1681314838912, max timestamp=1681314838946Num uniques 859; 2023-04-12 15:53:58,949 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39331, new segment data size=9561, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-12 15:53:58,950 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681314838912, max timestamp=1681314838946Num uniques 859; 2023-04-12 15:53:58,950 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39324, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-12 15:53:58,970 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:58,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58374 deadline: 1681314898964, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:58,977 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:58,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58414 deadline: 1681314898968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:58,980 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:58,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58340 deadline: 1681314898969, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:58,980 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:58,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58398 deadline: 1681314898971, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:58,985 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:53:58,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:53:58,987 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:53:58,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:53:58,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:53:58,989 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:53:58,989 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:53:58,989 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:53:58,989 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:53:58,990 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:58,990 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:58,991 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:58,991 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:53:58,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:53:58,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:53:58,992 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:58,992 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:53:58,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:58,992 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:58,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:53:58,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:58,993 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:53:58,993 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:53:58,993 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:53:58,993 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:53:58,993 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:53:58,993 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:53:58,996 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:53:58,996 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:53:58,996 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:58,996 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. started... 2023-04-12 15:53:58,996 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. as already flushing 2023-04-12 15:53:58,996 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:58,996 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:53:58,996 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:53:58,996 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:58,996 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:53:58,996 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:53:58,998 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:58,998 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:58,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:58,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:58,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:58,998 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:58,998 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:53:58,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:58,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:58,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:58,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:58,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:59,000 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:53:59,000 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7063e984[Count = 0] remaining members to acquire global barrier 2023-04-12 15:53:59,000 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:53:59,000 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:59,002 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:59,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:59,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:59,003 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:53:59,003 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:53:59,003 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:53:59,003 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:59,003 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:53:59,004 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:53:59,004 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:53:59,004 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:59,005 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:53:59,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:59,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:59,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:59,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:59,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:59,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:59,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:59,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:59,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:59,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:59,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:53:59,007 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:53:59,007 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:53:59,007 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:53:59,007 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:53:59,007 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:53:59,007 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:53:59,009 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:59,009 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:59,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:59,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:59,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:59,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:59,009 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:53:59,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:59,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:53:59,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:53:59,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:59,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:59,010 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:59,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:59,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:59,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:59,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:59,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:59,013 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:59,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:59,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:59,019 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:59,019 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:59,019 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:53:59,019 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:59,019 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:53:59,020 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:53:59,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:53:59,020 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:53:59,020 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:53:59,020 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:59,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:53:59,020 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:59,020 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:59,020 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:59,020 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:53:59,020 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:53:59,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:53:59,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:53:59,052 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:59,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58354 deadline: 1681314899049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:59,075 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:59,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58374 deadline: 1681314899072, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:59,078 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:59,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58414 deadline: 1681314899078, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:59,087 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:59,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58340 deadline: 1681314899081, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:59,090 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:59,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58398 deadline: 1681314899082, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:59,120 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:53:59,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:53:59,123 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:53:59,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:53:59,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:53:59,134 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:53:59,134 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:53:59,134 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:53:59,134 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:53:59,135 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:59,135 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:59,136 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:53:59,136 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:59,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:53:59,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:53:59,139 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:59,139 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:53:59,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:59,139 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:59,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:53:59,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:59,140 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:53:59,144 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:53:59,144 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:53:59,144 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:53:59,144 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:53:59,144 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:53:59,144 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:53:59,144 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:53:59,144 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:59,144 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. started... 2023-04-12 15:53:59,145 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. as already flushing 2023-04-12 15:53:59,145 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:59,145 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:53:59,145 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:53:59,145 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:59,145 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:53:59,145 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:53:59,147 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:59,147 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:59,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:59,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:59,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:59,147 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:59,147 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:53:59,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:59,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:59,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:59,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:59,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:59,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:53:59,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c16e3fd[Count = 0] remaining members to acquire global barrier 2023-04-12 15:53:59,149 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:53:59,149 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:59,150 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:59,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:59,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:59,150 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:53:59,150 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:53:59,150 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:53:59,151 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:59,151 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:53:59,153 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:59,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:59,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:59,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:59,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:59,154 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:53:59,154 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:53:59,154 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:53:59,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:59,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:59,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:59,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:59,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:59,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:59,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:53:59,157 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:53:59,157 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:53:59,157 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:53:59,157 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:53:59,157 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:53:59,157 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:53:59,160 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:59,160 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:59,160 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:53:59,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:59,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:59,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:59,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:59,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:59,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:53:59,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:53:59,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:59,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:59,162 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:59,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:59,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:59,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:59,163 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:59,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:59,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:59,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:59,166 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:53:59,166 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:53:59,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:53:59,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:53:59,166 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:59,167 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:59,167 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:59,167 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:59,167 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:59,167 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:59,167 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:59,167 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:53:59,167 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:53:59,167 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:53:59,167 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:53:59,167 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:53:59,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:53:59,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:53:59,167 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:53:59,259 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:59,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58354 deadline: 1681314899254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:59,267 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:53:59,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:53:59,270 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:53:59,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:53:59,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:53:59,271 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:53:59,271 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:53:59,271 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:53:59,271 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:53:59,271 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:59,272 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:59,273 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:53:59,273 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:59,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:53:59,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:53:59,273 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:59,273 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:53:59,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:59,274 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:59,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:53:59,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:59,274 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:53:59,274 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:53:59,274 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:53:59,275 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:53:59,275 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:53:59,275 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:53:59,275 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:53:59,275 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:53:59,275 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:59,275 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. started... 2023-04-12 15:53:59,275 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. as already flushing 2023-04-12 15:53:59,275 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:59,275 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:53:59,275 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:53:59,275 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:59,275 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:53:59,275 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:53:59,278 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:59,278 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:59,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:59,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:59,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:59,278 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:59,278 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:53:59,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:59,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:59,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:59,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:59,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:59,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:53:59,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@121b8056[Count = 0] remaining members to acquire global barrier 2023-04-12 15:53:59,280 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:53:59,280 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:59,281 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:59,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:59,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:59,282 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:53:59,282 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:53:59,282 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:53:59,282 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:59,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58374 deadline: 1681314899276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:59,284 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:59,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58414 deadline: 1681314899279, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:59,288 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:59,288 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:53:59,292 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:59,292 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:59,292 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:53:59,292 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:53:59,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:59,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:59,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:59,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58340 deadline: 1681314899289, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:59,292 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:53:59,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:59,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:59,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:59,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:59,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:59,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:59,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:59,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:53:59,295 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:53:59,295 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:53:59,295 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:53:59,295 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:53:59,295 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:53:59,295 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:53:59,297 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:59,297 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:59,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:59,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:59,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:59,297 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:53:59,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:59,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:59,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:53:59,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:53:59,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:59,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:59,297 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:59,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:59,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:59,298 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:59,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58398 deadline: 1681314899293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:59,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:59,299 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:59,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:59,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:59,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:59,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:59,303 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:59,303 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:59,303 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:59,303 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:59,303 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:59,303 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:59,303 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:53:59,303 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:53:59,303 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:59,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:53:59,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:53:59,303 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:53:59,303 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:53:59,303 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:53:59,303 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:53:59,303 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:53:59,303 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:53:59,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:53:59,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:53:59,305 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=626 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/A/54cea4ee52eb4d3385730f84924733a3 2023-04-12 15:53:59,321 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=626 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/B/8747c3c229ec4277881cd2644d77a4a5 2023-04-12 15:53:59,343 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=626 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/C/585d29c5d7d14cf4bf0f9e8fdef4aef5 2023-04-12 15:53:59,349 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/A/54cea4ee52eb4d3385730f84924733a3 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/54cea4ee52eb4d3385730f84924733a3 2023-04-12 15:53:59,355 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/54cea4ee52eb4d3385730f84924733a3, entries=250, sequenceid=626, filesize=16.7 K 2023-04-12 15:53:59,356 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/B/8747c3c229ec4277881cd2644d77a4a5 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/8747c3c229ec4277881cd2644d77a4a5 2023-04-12 15:53:59,361 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/8747c3c229ec4277881cd2644d77a4a5, entries=150, sequenceid=626, filesize=11.9 K 2023-04-12 15:53:59,362 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/C/585d29c5d7d14cf4bf0f9e8fdef4aef5 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/585d29c5d7d14cf4bf0f9e8fdef4aef5 2023-04-12 15:53:59,371 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/585d29c5d7d14cf4bf0f9e8fdef4aef5, entries=150, sequenceid=626, filesize=11.9 K 2023-04-12 15:53:59,372 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=118.14 KB/120976 for feda8274e3aaad09f0f3a1336e663d20 in 497ms, sequenceid=626, compaction requested=true 2023-04-12 15:53:59,373 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for feda8274e3aaad09f0f3a1336e663d20: 2023-04-12 15:53:59,373 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:53:59,373 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-12 15:53:59,373 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:53:59,373 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-12 15:53:59,373 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-12 15:53:59,376 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 90660 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-12 15:53:59,377 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1912): feda8274e3aaad09f0f3a1336e663d20/A is initiating minor compaction (all files) 2023-04-12 15:53:59,377 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of feda8274e3aaad09f0f3a1336e663d20/A in TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:59,377 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/98652c09fc6f47709fbb1ffcd0800b8e, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/538b0f0b3a6d478ab26459ed27598c7b, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/31b3c6bb09a8434596a142a1de704dad, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/62693f50f24345fe8e5c1672db71a03b, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/19892e6cb9534a64b07605bf9fa6deee, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/3b54c347241c4d0eb27e2ab352d1fb72, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/54cea4ee52eb4d3385730f84924733a3] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp, totalSize=88.5 K 2023-04-12 15:53:59,377 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 85780 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-12 15:53:59,377 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1912): feda8274e3aaad09f0f3a1336e663d20/B is initiating minor compaction (all files) 2023-04-12 15:53:59,377 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2259): Starting compaction of feda8274e3aaad09f0f3a1336e663d20/B in TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:59,378 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/44460f66245640c8835841e0145d0671, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/8be84eb2f3b1437a964861c66b5ee6ee, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/6dca3d7853cf4752829affa84c094274, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/6987831befee4dd7a95b462a1caab04c, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/ce27e115c43f48aba2e29720eaec8653, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/3ad20f89347e488bb98f491671452d33, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/8747c3c229ec4277881cd2644d77a4a5] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp, totalSize=83.8 K 2023-04-12 15:53:59,378 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 98652c09fc6f47709fbb1ffcd0800b8e, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=484, earliestPutTs=1681314830953 2023-04-12 15:53:59,378 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 538b0f0b3a6d478ab26459ed27598c7b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=520, earliestPutTs=1681314832254 2023-04-12 15:53:59,378 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 44460f66245640c8835841e0145d0671, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=484, earliestPutTs=1681314830953 2023-04-12 15:53:59,380 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 31b3c6bb09a8434596a142a1de704dad, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=533, earliestPutTs=1681314833388 2023-04-12 15:53:59,380 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 8be84eb2f3b1437a964861c66b5ee6ee, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=520, earliestPutTs=1681314832254 2023-04-12 15:53:59,380 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 62693f50f24345fe8e5c1672db71a03b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=568, earliestPutTs=1681314834085 2023-04-12 15:53:59,381 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 6dca3d7853cf4752829affa84c094274, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=533, earliestPutTs=1681314833388 2023-04-12 15:53:59,384 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 19892e6cb9534a64b07605bf9fa6deee, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=578, earliestPutTs=1681314834721 2023-04-12 15:53:59,385 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 6987831befee4dd7a95b462a1caab04c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=568, earliestPutTs=1681314834085 2023-04-12 15:53:59,385 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 3b54c347241c4d0eb27e2ab352d1fb72, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=614, earliestPutTs=1681314836549 2023-04-12 15:53:59,385 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting ce27e115c43f48aba2e29720eaec8653, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=578, earliestPutTs=1681314834721 2023-04-12 15:53:59,385 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 54cea4ee52eb4d3385730f84924733a3, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=626, earliestPutTs=1681314837718 2023-04-12 15:53:59,386 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 3ad20f89347e488bb98f491671452d33, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=614, earliestPutTs=1681314836549 2023-04-12 15:53:59,386 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 8747c3c229ec4277881cd2644d77a4a5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=626, earliestPutTs=1681314837718 2023-04-12 15:53:59,404 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:53:59,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:53:59,407 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:53:59,414 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] throttle.PressureAwareThroughputController(145): feda8274e3aaad09f0f3a1336e663d20#B#compaction#63 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:53:59,416 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] throttle.PressureAwareThroughputController(145): feda8274e3aaad09f0f3a1336e663d20#A#compaction#64 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:53:59,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:53:59,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:53:59,418 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:53:59,418 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:53:59,418 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:53:59,419 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:53:59,419 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:59,419 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:59,420 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:53:59,420 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:59,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:53:59,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:53:59,421 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:59,421 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:53:59,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:59,421 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:59,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:53:59,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:59,422 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:53:59,423 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:53:59,423 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:53:59,423 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:53:59,423 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:53:59,423 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:53:59,424 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:53:59,424 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:53:59,424 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:59,424 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. started... 2023-04-12 15:53:59,424 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing feda8274e3aaad09f0f3a1336e663d20 3/3 column families, dataSize=118.14 KB heapSize=334.46 KB 2023-04-12 15:53:59,425 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK feda8274e3aaad09f0f3a1336e663d20, store=A 2023-04-12 15:53:59,425 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:53:59,425 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK feda8274e3aaad09f0f3a1336e663d20, store=B 2023-04-12 15:53:59,425 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:53:59,425 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK feda8274e3aaad09f0f3a1336e663d20, store=C 2023-04-12 15:53:59,425 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:53:59,434 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/B/ca8e69fdbabd46b098c4fbc93f8d95b8 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/ca8e69fdbabd46b098c4fbc93f8d95b8 2023-04-12 15:53:59,439 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=660 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/A/90fbede993c741a5ae751c399cac4197 2023-04-12 15:53:59,442 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in feda8274e3aaad09f0f3a1336e663d20/B of feda8274e3aaad09f0f3a1336e663d20 into ca8e69fdbabd46b098c4fbc93f8d95b8(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:53:59,442 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for feda8274e3aaad09f0f3a1336e663d20: 2023-04-12 15:53:59,442 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20., storeName=feda8274e3aaad09f0f3a1336e663d20/B, priority=9, startTime=1681314839373; duration=0sec 2023-04-12 15:53:59,442 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:53:59,442 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-12 15:53:59,445 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 85780 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-12 15:53:59,446 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1912): feda8274e3aaad09f0f3a1336e663d20/C is initiating minor compaction (all files) 2023-04-12 15:53:59,446 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2259): Starting compaction of feda8274e3aaad09f0f3a1336e663d20/C in TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:59,446 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/dc35c07fb7994813b6248e58273954ed, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/b985bf1b49d94215a9efe19d9a8e1895, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/18a512a078e9487486d16cc19306fe24, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/b297403b25a042c39756052596873237, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/d40b89f36464453ab1d40cce50cc40ec, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/5e234b651bf54320b2e8581163dbdcef, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/585d29c5d7d14cf4bf0f9e8fdef4aef5] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp, totalSize=83.8 K 2023-04-12 15:53:59,447 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting dc35c07fb7994813b6248e58273954ed, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=484, earliestPutTs=1681314830953 2023-04-12 15:53:59,448 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting b985bf1b49d94215a9efe19d9a8e1895, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=520, earliestPutTs=1681314832254 2023-04-12 15:53:59,449 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 18a512a078e9487486d16cc19306fe24, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=533, earliestPutTs=1681314833388 2023-04-12 15:53:59,449 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting b297403b25a042c39756052596873237, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=568, earliestPutTs=1681314834085 2023-04-12 15:53:59,450 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting d40b89f36464453ab1d40cce50cc40ec, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=578, earliestPutTs=1681314834721 2023-04-12 15:53:59,451 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 5e234b651bf54320b2e8581163dbdcef, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=614, earliestPutTs=1681314836549 2023-04-12 15:53:59,452 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 585d29c5d7d14cf4bf0f9e8fdef4aef5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=626, earliestPutTs=1681314837718 2023-04-12 15:53:59,474 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=660 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/B/13a6fdb6301e4ed5a5a193a58a1f0e57 2023-04-12 15:53:59,476 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] throttle.PressureAwareThroughputController(145): feda8274e3aaad09f0f3a1336e663d20#C#compaction#67 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:53:59,490 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/C/96f48745721b45d18a4a09e4b0cfe292 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/96f48745721b45d18a4a09e4b0cfe292 2023-04-12 15:53:59,497 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in feda8274e3aaad09f0f3a1336e663d20/C of feda8274e3aaad09f0f3a1336e663d20 into 96f48745721b45d18a4a09e4b0cfe292(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:53:59,497 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for feda8274e3aaad09f0f3a1336e663d20: 2023-04-12 15:53:59,497 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20., storeName=feda8274e3aaad09f0f3a1336e663d20/C, priority=9, startTime=1681314839373; duration=0sec 2023-04-12 15:53:59,497 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:53:59,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on feda8274e3aaad09f0f3a1336e663d20 2023-04-12 15:53:59,561 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. as already flushing 2023-04-12 15:53:59,591 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:59,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58354 deadline: 1681314899589, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:59,596 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:59,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58374 deadline: 1681314899591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:59,597 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:59,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58340 deadline: 1681314899595, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:59,601 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:59,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58414 deadline: 1681314899598, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:59,605 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:59,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58398 deadline: 1681314899601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:59,699 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:59,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58354 deadline: 1681314899697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:59,846 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/A/9befdd333838463a914b86e2aa273175 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/9befdd333838463a914b86e2aa273175 2023-04-12 15:53:59,854 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in feda8274e3aaad09f0f3a1336e663d20/A of feda8274e3aaad09f0f3a1336e663d20 into 9befdd333838463a914b86e2aa273175(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:53:59,855 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for feda8274e3aaad09f0f3a1336e663d20: 2023-04-12 15:53:59,855 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20., storeName=feda8274e3aaad09f0f3a1336e663d20/A, priority=9, startTime=1681314839373; duration=0sec 2023-04-12 15:53:59,855 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:53:59,900 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=37.63 KB at sequenceid=660 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/C/3a2de69d594c4b3d9ebb6dd00f15f621 2023-04-12 15:53:59,901 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:53:59,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58354 deadline: 1681314899900, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:59,907 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/A/90fbede993c741a5ae751c399cac4197 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/90fbede993c741a5ae751c399cac4197 2023-04-12 15:53:59,914 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/90fbede993c741a5ae751c399cac4197, entries=150, sequenceid=660, filesize=11.9 K 2023-04-12 15:53:59,916 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/B/13a6fdb6301e4ed5a5a193a58a1f0e57 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/13a6fdb6301e4ed5a5a193a58a1f0e57 2023-04-12 15:53:59,922 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/13a6fdb6301e4ed5a5a193a58a1f0e57, entries=150, sequenceid=660, filesize=11.9 K 2023-04-12 15:53:59,923 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/C/3a2de69d594c4b3d9ebb6dd00f15f621 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/3a2de69d594c4b3d9ebb6dd00f15f621 2023-04-12 15:53:59,928 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/3a2de69d594c4b3d9ebb6dd00f15f621, entries=150, sequenceid=660, filesize=11.9 K 2023-04-12 15:53:59,929 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~118.14 KB/120976, heapSize ~334.41 KB/342440, currentSize=60.38 KB/61830 for feda8274e3aaad09f0f3a1336e663d20 in 505ms, sequenceid=660, compaction requested=false 2023-04-12 15:53:59,929 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for feda8274e3aaad09f0f3a1336e663d20: 2023-04-12 15:53:59,929 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:53:59,930 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:53:59,930 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:53:59,930 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:59,930 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:53:59,930 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:53:59,931 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:59,932 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:59,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:59,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:59,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:59,932 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:59,932 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:53:59,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:59,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:59,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:59,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:59,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:59,934 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:53:59,934 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@83178ce[Count = 0] remaining members to acquire global barrier 2023-04-12 15:53:59,934 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:53:59,934 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:59,935 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:59,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:59,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:59,935 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:53:59,935 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:53:59,935 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:53:59,936 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:59,936 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:53:59,938 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:59,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:59,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:59,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:59,938 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:53:59,939 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:53:59,939 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:53:59,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:59,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:59,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:59,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:59,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:59,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:59,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:59,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:53:59,941 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:53:59,941 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:53:59,941 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:53:59,941 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:53:59,941 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:53:59,942 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:53:59,944 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:59,944 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:59,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:59,944 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:53:59,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:59,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:59,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:53:59,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:53:59,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:53:59,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:53:59,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:53:59,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:59,946 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:59,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:59,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:53:59,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:59,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:59,948 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:59,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:53:59,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:53:59,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:59,950 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:53:59,950 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:59,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:53:59,950 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:53:59,951 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:53:59,950 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:53:59,950 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:59,951 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:53:59,951 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:53:59,951 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:53:59,951 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:53:59,950 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:53:59,951 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:59,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:53:59,951 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:53:59,951 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:53:59,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:53:59,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:00,051 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:00,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:00,053 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:00,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:00,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:00,056 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:00,056 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:00,057 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:00,057 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:00,057 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:00,057 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:00,062 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,062 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:00,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:00,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:00,063 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,063 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:00,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:00,063 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:00,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:00,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:00,064 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:00,064 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:00,065 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:00,065 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:00,065 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:00,065 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:00,065 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:00,065 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:00,065 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:54:00,065 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. started... 2023-04-12 15:54:00,065 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing feda8274e3aaad09f0f3a1336e663d20 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-12 15:54:00,065 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK feda8274e3aaad09f0f3a1336e663d20, store=A 2023-04-12 15:54:00,066 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:00,066 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK feda8274e3aaad09f0f3a1336e663d20, store=B 2023-04-12 15:54:00,066 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:00,066 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK feda8274e3aaad09f0f3a1336e663d20, store=C 2023-04-12 15:54:00,066 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:00,079 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=675 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/A/dd8970471d2a4331a8462a30975daad0 2023-04-12 15:54:00,099 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=675 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/B/666862bf676443dc9a35753f3b8e4389 2023-04-12 15:54:00,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on feda8274e3aaad09f0f3a1336e663d20 2023-04-12 15:54:00,100 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. as already flushing 2023-04-12 15:54:00,114 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=675 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/C/261fa644b7e540df93e142140c903b16 2023-04-12 15:54:00,129 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:00,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58398 deadline: 1681314900124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,130 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:00,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58340 deadline: 1681314900126, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,131 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:00,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58414 deadline: 1681314900128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,132 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/A/dd8970471d2a4331a8462a30975daad0 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/dd8970471d2a4331a8462a30975daad0 2023-04-12 15:54:00,133 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:00,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58374 deadline: 1681314900130, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,142 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/dd8970471d2a4331a8462a30975daad0, entries=150, sequenceid=675, filesize=11.9 K 2023-04-12 15:54:00,146 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/B/666862bf676443dc9a35753f3b8e4389 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/666862bf676443dc9a35753f3b8e4389 2023-04-12 15:54:00,152 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/666862bf676443dc9a35753f3b8e4389, entries=150, sequenceid=675, filesize=11.9 K 2023-04-12 15:54:00,153 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/C/261fa644b7e540df93e142140c903b16 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/261fa644b7e540df93e142140c903b16 2023-04-12 15:54:00,159 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/261fa644b7e540df93e142140c903b16, entries=150, sequenceid=675, filesize=11.9 K 2023-04-12 15:54:00,161 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for feda8274e3aaad09f0f3a1336e663d20 in 96ms, sequenceid=675, compaction requested=true 2023-04-12 15:54:00,161 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for feda8274e3aaad09f0f3a1336e663d20: 2023-04-12 15:54:00,161 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:54:00,161 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:00,161 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:00,161 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,164 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:00,164 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:00,166 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,166 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:00,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:00,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:00,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:00,167 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:00,167 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:00,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:00,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:00,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:00,169 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:00,169 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@67f19f85[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:00,169 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:00,169 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:00,170 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:00,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:00,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:00,171 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:00,171 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:00,171 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:00,171 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,171 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:00,172 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:00,172 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:00,172 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:00,173 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:00,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:00,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:00,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:00,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:00,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:00,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:00,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:00,178 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:00,178 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:00,178 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:00,178 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:00,178 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:00,178 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:00,181 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:00,181 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:00,181 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:00,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:00,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:00,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:00,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:00,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:00,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:00,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:00,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:00,183 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:00,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:00,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:00,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:00,184 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:00,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:00,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,187 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:00,187 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:00,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:00,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:00,188 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,188 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:00,188 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:00,188 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:00,188 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:00,188 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,188 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:00,188 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:00,188 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:00,188 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:00,188 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:00,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:00,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:00,188 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:00,188 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:00,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on feda8274e3aaad09f0f3a1336e663d20 2023-04-12 15:54:00,206 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing feda8274e3aaad09f0f3a1336e663d20 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-12 15:54:00,206 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK feda8274e3aaad09f0f3a1336e663d20, store=A 2023-04-12 15:54:00,206 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:54:00,206 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK feda8274e3aaad09f0f3a1336e663d20, store=B 2023-04-12 15:54:00,206 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:54:00,206 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK feda8274e3aaad09f0f3a1336e663d20, store=C 2023-04-12 15:54:00,206 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:54:00,237 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:00,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58374 deadline: 1681314900234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,237 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:00,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58340 deadline: 1681314900234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,238 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:00,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58414 deadline: 1681314900236, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,238 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:00,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58354 deadline: 1681314900236, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,242 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:00,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58398 deadline: 1681314900237, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,289 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:00,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:00,291 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:00,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:00,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:00,292 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:00,292 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:00,292 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:00,292 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:00,292 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:00,292 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:00,294 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,294 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:00,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:00,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:00,294 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,294 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:00,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:00,294 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:00,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:00,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:00,295 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:00,295 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:00,295 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:00,295 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:00,295 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:00,296 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:00,296 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:00,296 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:54:00,296 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. started... 2023-04-12 15:54:00,296 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. as already flushing 2023-04-12 15:54:00,296 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:54:00,296 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:00,296 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:00,296 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,296 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:00,296 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:00,298 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,298 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:00,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:00,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:00,298 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:00,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:00,299 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:00,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:00,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:00,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:00,300 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:00,300 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ce3818d[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:00,300 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:00,300 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:00,307 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:00,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:00,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:00,307 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:00,307 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:00,307 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:00,307 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,308 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:00,309 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,309 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:00,309 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:00,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:00,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:00,309 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:00,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:00,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:00,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:00,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:00,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:00,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:00,311 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:00,311 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:00,311 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:00,311 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:00,311 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:00,311 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:00,313 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:00,313 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:00,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:00,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:00,313 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:00,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:00,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:00,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:00,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:00,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:00,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:00,314 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:00,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:00,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:00,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:00,316 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:00,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:00,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,319 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:00,319 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:00,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:00,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:00,319 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:00,319 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:00,319 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:00,319 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,319 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:00,319 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:00,319 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,319 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:00,319 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:00,319 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:00,319 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:00,319 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:00,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:00,320 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:00,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:00,342 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:00,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58340 deadline: 1681314900338, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,343 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:00,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58414 deadline: 1681314900339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,343 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:00,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58354 deadline: 1681314900339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,343 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:00,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58398 deadline: 1681314900343, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,419 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:00,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:00,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:00,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:00,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:00,423 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:00,423 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:00,423 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:00,423 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:00,423 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:00,424 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:00,426 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,426 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:00,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:00,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:00,426 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,426 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:00,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:00,427 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:00,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:00,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:00,427 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:00,427 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:00,427 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:00,427 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:00,428 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:00,428 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:00,428 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:00,428 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:00,428 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:54:00,428 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. started... 2023-04-12 15:54:00,428 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. as already flushing 2023-04-12 15:54:00,428 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:54:00,428 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:00,428 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:00,428 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,428 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:00,428 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:00,430 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,430 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:00,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:00,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:00,430 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:00,430 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:00,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:00,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:00,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:00,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:00,432 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:00,432 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3234b777[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:00,432 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:00,432 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:00,433 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:00,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:00,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:00,433 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:00,433 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:00,433 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:00,433 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,433 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:00,435 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,435 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:00,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:00,435 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:00,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:00,435 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:00,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:00,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:00,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:00,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:00,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:00,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:00,437 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:00,437 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:00,437 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:00,437 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:00,437 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:00,437 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:00,439 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:00,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58374 deadline: 1681314900439, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,440 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:00,440 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:00,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:00,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:00,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:00,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:00,440 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:00,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:00,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:00,441 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:00,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:00,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:00,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:00,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:00,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:00,442 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:00,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:00,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,444 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,444 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:00,444 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:00,445 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:00,445 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:00,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:00,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:00,445 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,445 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:00,445 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:00,445 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:00,445 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:00,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:00,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:00,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:00,445 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:00,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:00,445 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:00,445 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:00,545 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:00,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:00,546 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:00,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58340 deadline: 1681314900544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,546 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:00,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58414 deadline: 1681314900544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,547 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:00,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58354 deadline: 1681314900545, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,548 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:00,548 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:00,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58398 deadline: 1681314900545, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:00,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:00,549 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:00,549 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:00,549 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:00,549 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:00,549 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:00,549 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:00,551 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,551 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:00,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:00,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:00,551 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,551 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:00,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:00,551 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:00,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:00,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:00,552 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:00,552 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:00,552 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:00,552 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:00,552 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:00,552 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:00,553 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:00,553 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:00,553 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:54:00,553 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. started... 2023-04-12 15:54:00,553 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. as already flushing 2023-04-12 15:54:00,553 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:54:00,553 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:00,553 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:00,553 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,553 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:00,553 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:00,555 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,555 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:00,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:00,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:00,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:00,555 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:00,555 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:00,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:00,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:00,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:00,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:00,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31404ef7[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:00,557 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:00,557 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:00,562 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:00,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:00,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:00,562 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:00,562 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:00,562 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,562 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:00,562 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:00,563 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:00,563 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,564 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:00,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:00,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:00,564 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:00,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:00,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:00,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:00,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:00,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:00,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:00,566 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:00,566 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:00,566 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:00,566 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:00,566 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:00,567 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:00,568 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:00,568 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:00,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:00,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:00,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:00,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:00,568 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:00,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:00,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:00,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:00,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:00,569 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:00,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:00,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:00,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:00,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,570 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:00,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:00,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,572 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,572 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:00,572 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:00,572 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:00,572 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:00,572 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:00,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:00,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:00,572 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:00,572 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:00,573 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:00,572 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,573 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:00,573 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:00,573 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:00,573 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:00,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:00,573 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:00,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:00,633 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=696 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/A/2f9b585667b141aa8d61aab9377043a9 2023-04-12 15:54:00,650 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=696 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/B/aec10d4ea66641c69af3beef4aa280f3 2023-04-12 15:54:00,662 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=696 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/C/77fc796ca1714261a869b33c6ce22f51 2023-04-12 15:54:00,668 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/A/2f9b585667b141aa8d61aab9377043a9 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/2f9b585667b141aa8d61aab9377043a9 2023-04-12 15:54:00,673 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/2f9b585667b141aa8d61aab9377043a9, entries=200, sequenceid=696, filesize=14.3 K 2023-04-12 15:54:00,673 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:00,674 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/B/aec10d4ea66641c69af3beef4aa280f3 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/aec10d4ea66641c69af3beef4aa280f3 2023-04-12 15:54:00,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:00,676 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:00,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:00,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:00,677 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:00,677 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:00,677 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:00,677 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:00,677 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:00,677 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:00,679 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/aec10d4ea66641c69af3beef4aa280f3, entries=150, sequenceid=696, filesize=11.9 K 2023-04-12 15:54:00,680 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:00,680 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:00,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:00,680 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,680 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:00,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:00,681 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:00,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:00,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:00,681 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:00,681 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:00,682 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:00,682 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:00,682 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:00,682 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:00,684 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:00,684 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:00,684 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:54:00,684 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. started... 2023-04-12 15:54:00,684 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. as already flushing 2023-04-12 15:54:00,684 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:54:00,684 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:00,684 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:00,684 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,684 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:00,684 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:00,686 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,686 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:00,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:00,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:00,686 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:00,686 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:00,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:00,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:00,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:00,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:00,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:00,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a26e2ed[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:00,688 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:00,688 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:00,689 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:00,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:00,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:00,689 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:00,689 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:00,689 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:00,690 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,690 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:00,691 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,691 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:00,691 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:00,691 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:00,691 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/C/77fc796ca1714261a869b33c6ce22f51 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/77fc796ca1714261a869b33c6ce22f51 2023-04-12 15:54:00,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:00,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:00,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:00,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:00,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:00,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:00,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:00,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:00,696 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:00,696 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:00,696 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:00,696 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:00,696 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:00,696 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:00,701 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/77fc796ca1714261a869b33c6ce22f51, entries=150, sequenceid=696, filesize=11.9 K 2023-04-12 15:54:00,702 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=53.67 KB/54960 for feda8274e3aaad09f0f3a1336e663d20 in 496ms, sequenceid=696, compaction requested=true 2023-04-12 15:54:00,702 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for feda8274e3aaad09f0f3a1336e663d20: 2023-04-12 15:54:00,702 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:54:00,702 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-12 15:54:00,702 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:54:00,702 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-12 15:54:00,702 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:54:00,704 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 51900 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-12 15:54:00,704 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1912): feda8274e3aaad09f0f3a1336e663d20/A is initiating minor compaction (all files) 2023-04-12 15:54:00,704 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of feda8274e3aaad09f0f3a1336e663d20/A in TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:54:00,704 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/9befdd333838463a914b86e2aa273175, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/90fbede993c741a5ae751c399cac4197, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/dd8970471d2a4331a8462a30975daad0, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/2f9b585667b141aa8d61aab9377043a9] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp, totalSize=50.7 K 2023-04-12 15:54:00,704 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49460 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-12 15:54:00,705 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1912): feda8274e3aaad09f0f3a1336e663d20/B is initiating minor compaction (all files) 2023-04-12 15:54:00,705 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 9befdd333838463a914b86e2aa273175, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=626, earliestPutTs=1681314837718 2023-04-12 15:54:00,705 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2259): Starting compaction of feda8274e3aaad09f0f3a1336e663d20/B in TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:54:00,705 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/ca8e69fdbabd46b098c4fbc93f8d95b8, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/13a6fdb6301e4ed5a5a193a58a1f0e57, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/666862bf676443dc9a35753f3b8e4389, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/aec10d4ea66641c69af3beef4aa280f3] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp, totalSize=48.3 K 2023-04-12 15:54:00,705 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 90fbede993c741a5ae751c399cac4197, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=660, earliestPutTs=1681314838963 2023-04-12 15:54:00,705 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting ca8e69fdbabd46b098c4fbc93f8d95b8, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=626, earliestPutTs=1681314837718 2023-04-12 15:54:00,705 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting dd8970471d2a4331a8462a30975daad0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=675, earliestPutTs=1681314839580 2023-04-12 15:54:00,706 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 13a6fdb6301e4ed5a5a193a58a1f0e57, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=660, earliestPutTs=1681314838963 2023-04-12 15:54:00,706 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 2f9b585667b141aa8d61aab9377043a9, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=696, earliestPutTs=1681314840123 2023-04-12 15:54:00,707 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 666862bf676443dc9a35753f3b8e4389, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=675, earliestPutTs=1681314839580 2023-04-12 15:54:00,707 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting aec10d4ea66641c69af3beef4aa280f3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=696, earliestPutTs=1681314840124 2023-04-12 15:54:00,708 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:00,708 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:00,708 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:00,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:00,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:00,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:00,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:00,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:00,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:00,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:00,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:00,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:00,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:00,710 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:00,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:00,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:00,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:00,712 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,715 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:00,715 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,715 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:00,715 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:00,715 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,715 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:00,715 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:00,715 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:00,715 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:00,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:00,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:00,715 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:00,715 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:00,716 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:00,716 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:00,716 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:00,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:00,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:00,722 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] throttle.PressureAwareThroughputController(145): feda8274e3aaad09f0f3a1336e663d20#A#compaction#75 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:54:00,723 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] throttle.PressureAwareThroughputController(145): feda8274e3aaad09f0f3a1336e663d20#B#compaction#76 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:54:00,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on feda8274e3aaad09f0f3a1336e663d20 2023-04-12 15:54:00,749 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing feda8274e3aaad09f0f3a1336e663d20 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-12 15:54:00,749 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK feda8274e3aaad09f0f3a1336e663d20, store=A 2023-04-12 15:54:00,750 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:00,750 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK feda8274e3aaad09f0f3a1336e663d20, store=B 2023-04-12 15:54:00,750 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:00,750 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK feda8274e3aaad09f0f3a1336e663d20, store=C 2023-04-12 15:54:00,750 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:00,752 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/A/8b1c4b3960d8446cae500b3c09331f73 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/8b1c4b3960d8446cae500b3c09331f73 2023-04-12 15:54:00,754 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/B/22b358b703e447949f97b66d09bfa35e as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/22b358b703e447949f97b66d09bfa35e 2023-04-12 15:54:00,762 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in feda8274e3aaad09f0f3a1336e663d20/A of feda8274e3aaad09f0f3a1336e663d20 into 8b1c4b3960d8446cae500b3c09331f73(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:54:00,762 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for feda8274e3aaad09f0f3a1336e663d20: 2023-04-12 15:54:00,762 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20., storeName=feda8274e3aaad09f0f3a1336e663d20/A, priority=12, startTime=1681314840702; duration=0sec 2023-04-12 15:54:00,762 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:54:00,763 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-12 15:54:00,765 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49460 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-12 15:54:00,765 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1912): feda8274e3aaad09f0f3a1336e663d20/C is initiating minor compaction (all files) 2023-04-12 15:54:00,765 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of feda8274e3aaad09f0f3a1336e663d20/C in TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:54:00,765 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/96f48745721b45d18a4a09e4b0cfe292, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/3a2de69d594c4b3d9ebb6dd00f15f621, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/261fa644b7e540df93e142140c903b16, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/77fc796ca1714261a869b33c6ce22f51] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp, totalSize=48.3 K 2023-04-12 15:54:00,766 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 96f48745721b45d18a4a09e4b0cfe292, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=626, earliestPutTs=1681314837718 2023-04-12 15:54:00,766 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 3a2de69d594c4b3d9ebb6dd00f15f621, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=660, earliestPutTs=1681314838963 2023-04-12 15:54:00,768 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 261fa644b7e540df93e142140c903b16, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=675, earliestPutTs=1681314839580 2023-04-12 15:54:00,769 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 77fc796ca1714261a869b33c6ce22f51, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=696, earliestPutTs=1681314840124 2023-04-12 15:54:00,772 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in feda8274e3aaad09f0f3a1336e663d20/B of feda8274e3aaad09f0f3a1336e663d20 into 22b358b703e447949f97b66d09bfa35e(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:54:00,772 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for feda8274e3aaad09f0f3a1336e663d20: 2023-04-12 15:54:00,772 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20., storeName=feda8274e3aaad09f0f3a1336e663d20/B, priority=12, startTime=1681314840702; duration=0sec 2023-04-12 15:54:00,773 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:54:00,787 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] throttle.PressureAwareThroughputController(145): feda8274e3aaad09f0f3a1336e663d20#C#compaction#78 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:54:00,810 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:00,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58374 deadline: 1681314900809, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,816 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:00,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:00,818 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:00,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:00,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:00,819 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:00,819 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:00,819 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:00,819 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:00,820 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:00,820 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:00,821 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,821 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:00,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:00,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:00,821 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,821 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:00,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:00,822 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:00,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:00,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:00,822 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:00,822 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:00,822 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:00,822 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:00,823 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:00,823 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:00,823 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:00,823 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:00,823 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:54:00,823 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. started... 2023-04-12 15:54:00,823 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. as already flushing 2023-04-12 15:54:00,823 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:54:00,823 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:00,823 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:00,823 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,823 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:00,823 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:00,824 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,824 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:00,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:00,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:00,825 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:00,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:00,825 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:00,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:00,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:00,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:00,827 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:00,827 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a29ffcd[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:00,827 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:00,827 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:00,834 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:00,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:00,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:00,834 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:00,834 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:00,834 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,834 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:00,834 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:00,836 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,836 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:00,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:00,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:00,836 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:00,836 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:00,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:00,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:00,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:00,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:00,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:00,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:00,840 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:00,840 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:00,840 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:00,840 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:00,840 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:00,840 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:00,842 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:00,842 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:00,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:00,842 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:00,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:00,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:00,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:00,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:00,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:00,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:00,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:00,843 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:00,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:00,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:00,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:00,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,847 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:00,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:00,851 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:00,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58414 deadline: 1681314900849, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,852 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:00,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58398 deadline: 1681314900849, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,858 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:00,858 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:00,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58340 deadline: 1681314900856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58354 deadline: 1681314900856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,860 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,860 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:00,861 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:00,861 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:00,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:00,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:00,861 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:00,861 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:00,861 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:00,861 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,861 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:00,861 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:00,861 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:00,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:00,861 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:00,861 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:00,861 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:00,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:00,914 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:00,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58374 deadline: 1681314900911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,962 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:00,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:00,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:00,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:00,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:00,965 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:00,965 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:00,965 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:00,965 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:00,965 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:00,965 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:00,968 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:00,968 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:00,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:00,968 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,968 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:00,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:00,968 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:00,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:00,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:00,969 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:00,969 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:00,969 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:00,969 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:00,969 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:00,969 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:00,970 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:00,970 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:00,970 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:54:00,970 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. started... 2023-04-12 15:54:00,970 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. as already flushing 2023-04-12 15:54:00,970 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:54:00,970 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:00,970 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:00,970 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,970 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:00,970 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:00,972 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,972 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:00,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:00,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:00,972 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:00,973 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:00,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:00,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:00,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:00,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:00,974 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:00,974 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d3178f0[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:00,974 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:00,974 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:00,975 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:00,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:00,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:00,975 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:00,975 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:00,975 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:00,976 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,976 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:00,981 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,981 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:00,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:00,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:00,981 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:00,981 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:00,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:00,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:00,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:00,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:00,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:00,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:00,984 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:00,984 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:00,984 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:00,984 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:00,984 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:00,984 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:00,986 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:00,986 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:00,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:00,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:00,986 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:00,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:00,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:00,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:00,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:00,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:00,986 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:00,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:00,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:00,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:00,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:00,988 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:00,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:00,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,990 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,990 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:00,990 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:00,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:00,990 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:00,990 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:00,990 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:00,990 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:00,990 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:00,990 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:00,990 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:00,990 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:00,991 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:00,990 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:00,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:00,991 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:00,997 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:00,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:00,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:01,091 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:01,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:01,093 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:01,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:01,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:01,094 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:01,094 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:01,094 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:01,094 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:01,095 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:01,095 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:01,098 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:01,098 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:01,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:01,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:01,104 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:01,104 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,104 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:01,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:01,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:01,104 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:01,105 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:01,105 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:01,105 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:01,105 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:01,105 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:01,105 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:01,105 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:01,105 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:54:01,105 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. started... 2023-04-12 15:54:01,106 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. as already flushing 2023-04-12 15:54:01,106 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:54:01,106 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:01,106 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:01,106 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,106 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:01,106 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:01,108 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:01,108 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:01,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:01,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:01,117 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:01,117 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:01,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:01,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:01,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:01,118 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:01,118 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1eb5ac57[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:01,118 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:01,118 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:01,118 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:01,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58374 deadline: 1681314901116, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,119 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:01,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:01,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:01,120 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:01,120 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:01,120 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:01,120 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,120 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:01,121 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,121 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:01,121 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:01,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:01,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:01,121 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:01,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:01,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:01,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:01,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:01,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:01,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:01,123 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:01,123 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:01,124 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:01,124 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:01,124 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:01,124 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:01,126 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:01,126 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:01,126 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:01,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:01,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:01,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:01,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:01,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:01,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:01,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:01,126 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:01,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:01,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:01,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:01,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:01,128 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:01,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:01,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,130 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:01,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:01,130 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:01,130 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:01,130 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:01,130 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:01,131 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:01,130 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:01,131 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,131 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:01,131 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:01,131 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:01,131 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:01,131 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:01,131 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:01,132 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:01,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:01,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:01,173 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=708 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/A/2e57c5566bfb489f9c4174afe49e367e 2023-04-12 15:54:01,207 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/C/13c7196fba4249949d37fa20ea528ea8 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/13c7196fba4249949d37fa20ea528ea8 2023-04-12 15:54:01,213 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in feda8274e3aaad09f0f3a1336e663d20/C of feda8274e3aaad09f0f3a1336e663d20 into 13c7196fba4249949d37fa20ea528ea8(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:54:01,213 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for feda8274e3aaad09f0f3a1336e663d20: 2023-04-12 15:54:01,213 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20., storeName=feda8274e3aaad09f0f3a1336e663d20/C, priority=12, startTime=1681314840702; duration=0sec 2023-04-12 15:54:01,213 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:54:01,231 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:01,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:01,234 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:01,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:01,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:01,236 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:01,236 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:01,236 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:01,236 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:01,238 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:01,238 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:01,244 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:01,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:01,244 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:01,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:01,244 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,244 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:01,245 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:01,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:01,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:01,245 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:01,246 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:01,246 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:01,246 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:01,246 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:01,246 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:01,248 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:01,248 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:01,248 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:54:01,248 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. started... 2023-04-12 15:54:01,248 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. as already flushing 2023-04-12 15:54:01,248 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:54:01,248 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:01,248 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:01,248 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,248 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:01,248 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:01,250 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:01,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:01,251 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:01,251 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:01,251 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:01,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:01,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:01,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:01,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:01,253 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:01,253 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@736914e6[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:01,253 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:01,253 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:01,256 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:01,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:01,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:01,256 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:01,256 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:01,256 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:01,256 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,256 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:01,258 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:01,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:01,258 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:01,258 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:01,258 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:01,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:01,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:01,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:01,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:01,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:01,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:01,262 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:01,262 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:01,262 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:01,262 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:01,262 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:01,262 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:01,265 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:01,265 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:01,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:01,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:01,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:01,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:01,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:01,265 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:01,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:01,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:01,266 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:01,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:01,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:01,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:01,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:01,267 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:01,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:01,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,271 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,271 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:01,271 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:01,271 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:01,271 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:01,271 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:01,271 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:01,271 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:01,271 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,271 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:01,271 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:01,271 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:01,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:01,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:01,271 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:01,272 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:01,272 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:01,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:01,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:01,356 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:01,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58414 deadline: 1681314901355, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,358 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:01,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58398 deadline: 1681314901357, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,363 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:01,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58340 deadline: 1681314901362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,363 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:01,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58354 deadline: 1681314901363, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,372 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:01,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:01,375 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:01,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:01,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:01,377 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:01,377 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:01,377 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:01,377 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:01,378 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:01,378 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:01,380 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:01,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:01,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:01,380 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:01,381 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,381 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:01,381 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:01,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:01,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:01,382 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:01,382 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:01,382 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:01,382 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:01,382 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:01,382 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:01,384 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:01,384 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:01,384 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:54:01,384 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. started... 2023-04-12 15:54:01,385 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. as already flushing 2023-04-12 15:54:01,386 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:54:01,386 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:01,386 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:01,386 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,386 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:01,386 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:01,388 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:01,388 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:01,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:01,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:01,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:01,395 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:01,395 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:01,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:01,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:01,396 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:01,396 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6cb4ad1[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:01,396 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:01,396 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:01,398 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:01,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:01,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:01,398 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:01,398 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:01,398 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:01,398 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,398 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:01,400 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:01,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:01,404 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:01,404 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:01,404 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:01,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:01,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:01,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:01,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:01,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:01,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:01,411 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:01,411 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:01,411 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:01,411 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:01,411 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:01,411 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:01,414 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:01,414 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:01,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:01,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:01,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:01,414 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:01,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:01,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:01,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:01,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:01,415 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:01,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:01,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:01,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:01,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:01,416 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:01,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:01,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,421 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,421 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:01,421 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:01,421 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,421 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:01,421 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:01,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:01,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:01,421 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:01,421 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:01,421 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:01,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:01,421 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:01,421 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:01,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:01,421 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:01,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:01,422 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:01,422 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:01,423 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:01,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58374 deadline: 1681314901421, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,522 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:01,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:01,525 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:01,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:01,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:01,526 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:01,526 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:01,526 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:01,526 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:01,526 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:01,526 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:01,527 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,527 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:01,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:01,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:01,528 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,528 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:01,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:01,528 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:01,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:01,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:01,529 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:01,529 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:01,529 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:01,529 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:01,529 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:01,529 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:01,529 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:01,529 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:01,529 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:54:01,529 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. started... 2023-04-12 15:54:01,529 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. as already flushing 2023-04-12 15:54:01,529 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:54:01,529 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:01,529 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:01,529 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,529 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:01,529 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:01,531 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,531 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:01,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:01,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:01,531 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:01,531 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:01,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:01,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:01,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:01,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:01,533 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:01,533 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@201b5400[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:01,533 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:01,533 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:01,535 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:01,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:01,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:01,535 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:01,535 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:01,535 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:01,535 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,535 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:01,536 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,536 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:01,536 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:01,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:01,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:01,536 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:01,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:01,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:01,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:01,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:01,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:01,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:01,539 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:01,539 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:01,539 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:01,539 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:01,539 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:01,539 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:01,541 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:01,541 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:01,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:01,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:01,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:01,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:01,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:01,541 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:01,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:01,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:01,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:01,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:01,543 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:01,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:01,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:01,544 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:01,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:01,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,546 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,546 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:01,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:01,546 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:01,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:01,547 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:01,547 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:01,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:01,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:01,547 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:01,547 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,547 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:01,547 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:01,547 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:01,547 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:01,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:01,547 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:01,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:01,588 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=708 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/B/ea5502a520744fa98b5417e8e8c5954c 2023-04-12 15:54:01,647 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:01,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:01,652 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:01,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:01,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:01,654 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:01,654 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:01,654 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:01,654 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:01,654 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:01,654 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:01,657 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,657 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:01,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:01,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:01,658 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,658 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:01,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:01,659 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:01,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:01,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:01,660 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:01,661 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:01,661 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:01,661 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:01,661 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:01,661 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:01,661 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:01,661 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:01,661 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:54:01,661 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. started... 2023-04-12 15:54:01,661 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. as already flushing 2023-04-12 15:54:01,661 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:54:01,661 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:01,661 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:01,661 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,661 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:01,662 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:01,668 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,668 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:01,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:01,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:01,668 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:01,668 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:01,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:01,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:01,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:01,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:01,670 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:01,670 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@137f8340[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:01,670 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:01,670 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:01,671 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:01,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:01,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:01,671 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:01,671 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:01,671 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,671 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:01,671 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:01,673 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,673 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:01,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:01,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:01,673 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:01,673 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:01,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:01,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:01,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:01,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:01,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:01,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:01,677 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:01,677 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:01,677 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:01,677 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:01,677 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:01,677 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:01,679 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:01,679 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:01,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:01,679 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:01,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:01,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:01,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:01,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:01,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:01,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:01,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:01,680 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:01,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:01,681 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:01,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:01,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:01,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:01,683 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,683 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:01,683 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:01,683 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:01,683 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:01,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:01,683 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,683 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:01,683 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:01,683 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:01,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:01,684 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:01,683 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:01,684 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:01,684 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:01,684 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:01,684 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:01,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:01,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:01,784 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:01,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:01,786 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:01,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:01,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:01,787 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:01,787 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:01,787 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:01,787 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:01,788 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:01,788 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:01,789 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,789 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:01,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:01,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:01,789 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,789 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:01,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:01,790 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:01,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:01,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:01,790 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:01,790 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:01,790 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:01,791 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:01,791 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:01,791 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:01,791 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:01,791 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:01,791 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:54:01,791 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. started... 2023-04-12 15:54:01,791 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. as already flushing 2023-04-12 15:54:01,791 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:54:01,791 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:01,791 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:01,791 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,791 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:01,791 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:01,792 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:01,792 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:01,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:01,793 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:01,793 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:01,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:01,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:01,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:01,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:01,794 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:01,794 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@280a6bed[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:01,794 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:01,794 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:01,797 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:01,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:01,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:01,797 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:01,797 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:01,797 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:01,797 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,797 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:01,798 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:01,798 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,798 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:01,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:01,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:01,798 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:01,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:01,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:01,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:01,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:01,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:01,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:01,802 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:01,802 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:01,802 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:01,802 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:01,802 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:01,802 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:01,803 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:01,803 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:01,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:01,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:01,803 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:01,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:01,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:01,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:01,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:01,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:01,804 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:01,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:01,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:01,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:01,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:01,806 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:01,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:01,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,811 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,811 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:01,811 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:01,811 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:01,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:01,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:01,812 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:01,811 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:01,811 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:01,812 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:01,812 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:01,812 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:01,812 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:01,812 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:01,812 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:01,812 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:01,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:01,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:01,912 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:01,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:01,915 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:01,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:01,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:01,917 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:01,917 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:01,917 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:01,917 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:01,918 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:01,918 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:01,919 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,919 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:01,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:01,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:01,919 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,919 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:01,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:01,920 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:01,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:01,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:01,921 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:01,922 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:01,922 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:01,922 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:01,922 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:01,922 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:01,922 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:01,922 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:01,922 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:54:01,922 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. started... 2023-04-12 15:54:01,923 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. as already flushing 2023-04-12 15:54:01,923 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:54:01,923 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:01,923 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:01,923 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,923 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:01,923 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:01,924 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:01,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:01,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:01,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:01,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:01,926 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:01,926 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:01,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58374 deadline: 1681314901926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,926 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:01,926 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:01,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:01,927 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:01,927 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@744b21ff[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:01,927 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:01,927 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:01,933 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:01,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:01,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:01,933 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:01,933 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:01,933 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:01,933 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,933 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:01,937 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,937 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:01,937 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:01,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:01,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:01,937 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:01,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:01,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:01,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:01,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:01,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:01,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:01,946 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:01,947 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:01,947 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:01,947 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:01,947 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:01,947 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:01,949 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:01,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:01,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:01,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:01,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:01,949 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:01,950 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:01,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:01,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:01,950 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:01,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:01,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:01,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:01,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:01,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:01,951 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:01,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:01,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,953 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,953 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:01,953 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:01,953 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:01,953 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:01,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:01,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:01,953 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:01,954 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:01,953 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:01,954 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:01,954 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:01,954 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:01,954 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:01,959 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:01,959 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:01,959 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:01,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:01,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:02,002 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=708 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/C/8896c8f0335045debdf87433b8682b0c 2023-04-12 15:54:02,008 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/A/2e57c5566bfb489f9c4174afe49e367e as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/2e57c5566bfb489f9c4174afe49e367e 2023-04-12 15:54:02,013 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/2e57c5566bfb489f9c4174afe49e367e, entries=200, sequenceid=708, filesize=14.3 K 2023-04-12 15:54:02,014 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/B/ea5502a520744fa98b5417e8e8c5954c as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/ea5502a520744fa98b5417e8e8c5954c 2023-04-12 15:54:02,019 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/ea5502a520744fa98b5417e8e8c5954c, entries=150, sequenceid=708, filesize=11.9 K 2023-04-12 15:54:02,020 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/C/8896c8f0335045debdf87433b8682b0c as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/8896c8f0335045debdf87433b8682b0c 2023-04-12 15:54:02,025 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/8896c8f0335045debdf87433b8682b0c, entries=150, sequenceid=708, filesize=11.9 K 2023-04-12 15:54:02,026 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for feda8274e3aaad09f0f3a1336e663d20 in 1277ms, sequenceid=708, compaction requested=false 2023-04-12 15:54:02,026 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for feda8274e3aaad09f0f3a1336e663d20: 2023-04-12 15:54:02,059 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:02,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:02,062 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:02,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:02,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:02,063 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:02,063 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:02,063 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:02,063 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:02,064 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:02,064 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:02,066 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:02,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:02,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:02,066 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:02,067 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,067 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:02,067 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:02,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:02,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:02,068 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:02,068 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:02,068 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:02,069 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:02,069 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:02,069 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:02,084 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:02,084 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:02,084 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:54:02,084 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. started... 2023-04-12 15:54:02,084 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing feda8274e3aaad09f0f3a1336e663d20 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-12 15:54:02,084 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK feda8274e3aaad09f0f3a1336e663d20, store=A 2023-04-12 15:54:02,085 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:02,085 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK feda8274e3aaad09f0f3a1336e663d20, store=B 2023-04-12 15:54:02,085 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:02,085 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK feda8274e3aaad09f0f3a1336e663d20, store=C 2023-04-12 15:54:02,085 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:02,097 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=731 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/A/3a432ad713e8410cbe92336938680cf9 2023-04-12 15:54:02,114 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=731 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/B/b509560954da41f28f642c7ba14b613a 2023-04-12 15:54:02,130 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=731 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/C/dd5762991a904e338ca505ec61a46c97 2023-04-12 15:54:02,134 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/A/3a432ad713e8410cbe92336938680cf9 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/3a432ad713e8410cbe92336938680cf9 2023-04-12 15:54:02,140 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/3a432ad713e8410cbe92336938680cf9, entries=150, sequenceid=731, filesize=11.9 K 2023-04-12 15:54:02,141 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/B/b509560954da41f28f642c7ba14b613a as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/b509560954da41f28f642c7ba14b613a 2023-04-12 15:54:02,148 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/b509560954da41f28f642c7ba14b613a, entries=150, sequenceid=731, filesize=11.9 K 2023-04-12 15:54:02,149 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/C/dd5762991a904e338ca505ec61a46c97 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/dd5762991a904e338ca505ec61a46c97 2023-04-12 15:54:02,154 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/dd5762991a904e338ca505ec61a46c97, entries=150, sequenceid=731, filesize=11.9 K 2023-04-12 15:54:02,155 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=0 B/0 for feda8274e3aaad09f0f3a1336e663d20 in 71ms, sequenceid=731, compaction requested=true 2023-04-12 15:54:02,155 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for feda8274e3aaad09f0f3a1336e663d20: 2023-04-12 15:54:02,155 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:54:02,155 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:02,155 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:02,155 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,155 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:02,155 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:02,157 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,157 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:02,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:02,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:02,157 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:02,157 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:02,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:02,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:02,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:02,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:02,159 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:02,159 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e8ac022[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:02,159 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:02,159 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:02,161 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:02,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:02,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:02,161 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:02,161 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:02,161 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:02,161 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,161 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:02,162 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:02,162 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:02,162 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:02,163 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:02,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:02,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:02,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:02,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:02,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:02,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:02,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:02,166 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:02,166 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:02,166 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:02,166 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:02,166 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:02,166 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:02,168 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:02,168 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:02,168 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:02,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:02,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:02,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:02,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:02,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:02,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:02,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:02,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:02,169 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:02,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:02,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:02,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:02,171 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:02,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:02,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,173 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,173 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:02,173 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:02,173 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:02,173 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:02,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:02,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:02,173 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:02,173 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,173 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:02,173 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:02,173 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:02,173 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:02,174 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:02,173 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:02,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:02,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:02,174 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:02,174 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:02,274 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:02,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:02,276 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:02,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:02,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:02,277 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:02,277 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:02,278 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:02,278 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:02,278 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:02,278 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:02,280 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,280 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:02,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:02,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:02,280 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,280 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:02,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:02,280 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:02,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:02,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:02,281 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:02,281 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:02,281 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:02,281 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:02,281 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:02,281 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:02,281 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:02,281 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:02,282 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:54:02,282 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. started... 2023-04-12 15:54:02,282 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for feda8274e3aaad09f0f3a1336e663d20: 2023-04-12 15:54:02,282 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:54:02,282 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:02,282 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:02,282 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,282 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:02,282 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:02,284 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:02,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:02,284 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:02,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:02,284 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:02,284 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:02,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:02,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:02,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:02,286 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:02,286 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61002730[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:02,286 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:02,286 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:02,288 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:02,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:02,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:02,288 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:02,288 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:02,288 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:02,288 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,288 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:02,290 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:02,290 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,290 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:02,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:02,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:02,290 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:02,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:02,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:02,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:02,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:02,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:02,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:02,293 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:02,293 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:02,293 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:02,293 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:02,293 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:02,293 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:02,295 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:02,295 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:02,295 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:02,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:02,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:02,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:02,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:02,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:02,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:02,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:02,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:02,299 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:02,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:02,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:02,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:02,300 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:02,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:02,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,303 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:02,303 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:02,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:02,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:02,303 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,303 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:02,303 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:02,303 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:02,303 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,303 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:02,303 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:02,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:02,303 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:02,304 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:02,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:02,304 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:02,304 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:02,304 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:02,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on feda8274e3aaad09f0f3a1336e663d20 2023-04-12 15:54:02,371 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing feda8274e3aaad09f0f3a1336e663d20 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-12 15:54:02,372 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK feda8274e3aaad09f0f3a1336e663d20, store=A 2023-04-12 15:54:02,372 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:02,372 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK feda8274e3aaad09f0f3a1336e663d20, store=B 2023-04-12 15:54:02,372 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:02,372 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK feda8274e3aaad09f0f3a1336e663d20, store=C 2023-04-12 15:54:02,372 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:02,389 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:02,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58354 deadline: 1681314902386, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,390 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=409, cellSize=18725, totalHeapSize=42776, min timestamp=1681314842374, max timestamp=1681314842388Num uniques 859; 2023-04-12 15:54:02,390 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=18725, suffix heap size=117064,new segment heap size=42776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=409 2023-04-12 15:54:02,390 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=409, cellSize=18725, totalHeapSize=42776, min timestamp=1681314842374, max timestamp=1681314842388Num uniques 859; 2023-04-12 15:54:02,390 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=18725, suffix heap size=117064,new segment heap size=42776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=409 2023-04-12 15:54:02,390 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=409, cellSize=18725, totalHeapSize=42776, min timestamp=1681314842374, max timestamp=1681314842388Num uniques 859; 2023-04-12 15:54:02,390 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=18725, suffix heap size=117064,new segment heap size=42776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=409 2023-04-12 15:54:02,404 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:02,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:02,406 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:02,407 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:02,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58340 deadline: 1681314902404, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:02,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:02,408 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:02,408 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:02,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58414 deadline: 1681314902404, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,408 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:02,408 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:02,408 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:02,409 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:02,409 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:02,408 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:02,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58398 deadline: 1681314902406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,414 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:02,414 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:02,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:02,415 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,415 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:02,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:02,415 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:02,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:02,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:02,415 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:02,416 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:02,416 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:02,416 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:02,416 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:02,416 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:02,418 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:02,418 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:02,418 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:54:02,418 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. started... 2023-04-12 15:54:02,418 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. as already flushing 2023-04-12 15:54:02,418 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:54:02,418 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:02,418 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:02,418 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,418 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:02,418 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:02,420 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:02,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:02,420 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:02,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:02,420 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:02,420 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:02,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:02,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:02,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:02,422 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:02,422 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1aae6dfe[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:02,422 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:02,422 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:02,423 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:02,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:02,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:02,423 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:02,423 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:02,423 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:02,423 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,424 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:02,426 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,426 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:02,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:02,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:02,426 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:02,426 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:02,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:02,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:02,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:02,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:02,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:02,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:02,428 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:02,428 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:02,428 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:02,428 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:02,428 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:02,428 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:02,431 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:02,431 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:02,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:02,431 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:02,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:02,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:02,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:02,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:02,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:02,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:02,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:02,432 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:02,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:02,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:02,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:02,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,433 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:02,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:02,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,436 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:02,436 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,436 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:02,436 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:02,436 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,436 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:02,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:02,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:02,436 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:02,437 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:02,437 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:02,437 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:02,437 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:02,437 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:02,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:02,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:02,437 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:02,437 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:02,491 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:02,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58354 deadline: 1681314902491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,509 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:02,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58340 deadline: 1681314902509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,512 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:02,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58414 deadline: 1681314902511, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,515 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:02,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58398 deadline: 1681314902512, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,537 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:02,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:02,540 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:02,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:02,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:02,542 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:02,542 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:02,542 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:02,542 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:02,544 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:02,544 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:02,545 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,545 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:02,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:02,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:02,546 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,546 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:02,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:02,547 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:02,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:02,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:02,547 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:02,547 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:02,547 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:02,548 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:02,548 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:02,548 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:02,552 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:02,552 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:02,552 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:54:02,552 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. started... 2023-04-12 15:54:02,552 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. as already flushing 2023-04-12 15:54:02,552 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:54:02,552 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:02,552 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:02,552 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,552 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:02,552 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:02,555 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,555 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:02,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:02,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:02,555 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:02,555 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:02,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:02,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:02,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:02,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:02,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:02,557 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:02,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37c7f38d[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:02,557 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:02,558 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:02,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:02,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:02,558 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,558 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:02,558 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:02,558 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:02,558 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:02,559 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,559 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:02,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:02,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:02,560 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:02,560 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:02,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:02,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:02,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:02,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:02,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:02,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:02,562 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:02,562 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:02,562 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:02,562 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:02,562 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:02,562 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:02,564 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:02,564 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:02,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:02,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:02,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:02,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:02,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:02,564 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:02,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:02,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:02,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:02,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:02,566 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:02,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:02,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:02,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,567 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:02,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:02,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,570 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,570 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:02,570 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:02,570 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:02,570 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:02,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:02,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:02,570 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:02,570 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:02,570 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:02,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:02,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:02,571 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,571 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:02,571 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:02,571 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:02,571 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:02,571 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:02,671 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:02,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:02,674 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:02,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:02,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:02,675 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:02,675 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:02,675 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:02,675 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:02,677 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:02,677 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:02,679 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:02,679 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:02,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:02,679 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,680 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:02,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:02,680 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:02,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:02,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:02,681 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:02,692 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:02,692 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:02,692 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:02,692 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:02,692 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:02,692 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:02,692 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:02,692 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:54:02,693 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. started... 2023-04-12 15:54:02,693 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. as already flushing 2023-04-12 15:54:02,693 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:54:02,693 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:02,693 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:02,693 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,693 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:02,693 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:02,694 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:02,694 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:02,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:02,695 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:02,695 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:02,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:02,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:02,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:02,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:02,696 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:02,696 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:02,696 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f14ae87[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:02,696 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:02,696 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:02,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58354 deadline: 1681314902693, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,697 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:02,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:02,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:02,697 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:02,698 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:02,698 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:02,698 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,698 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:02,700 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:02,700 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:02,700 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,700 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:02,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:02,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:02,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:02,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:02,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:02,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:02,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:02,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:02,703 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:02,703 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:02,703 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:02,703 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:02,703 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:02,703 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:02,704 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:02,704 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:02,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:02,704 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:02,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:02,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:02,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:02,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:02,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:02,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:02,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:02,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:02,706 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:02,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:02,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:02,708 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:02,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:02,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,713 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:02,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58340 deadline: 1681314902713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,714 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,715 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:02,715 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:02,715 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:02,715 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:02,715 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:02,715 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,715 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:02,715 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:02,715 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:02,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:02,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:02,715 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:02,715 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:02,715 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:02,715 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:02,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:02,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:02,717 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:02,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58414 deadline: 1681314902714, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,721 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:02,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58398 deadline: 1681314902717, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,782 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=743 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/A/2366452ecfbd4c69984d3cbe534c4e92 2023-04-12 15:54:02,811 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=743 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/B/864016bf332c4f76b473a25ed5ec6977 2023-04-12 15:54:02,815 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:02,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:02,819 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:02,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:02,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:02,824 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:02,824 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:02,824 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:02,824 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:02,825 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:02,825 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:02,826 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:02,826 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:02,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:02,827 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,827 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:02,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:02,827 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:02,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:02,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:02,828 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:02,829 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:02,830 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:02,830 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:02,830 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:02,830 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:02,836 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:02,836 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:02,836 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:54:02,836 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. started... 2023-04-12 15:54:02,836 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. as already flushing 2023-04-12 15:54:02,836 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:54:02,836 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:02,836 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:02,836 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,837 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:02,837 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:02,840 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,840 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:02,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:02,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:02,840 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:02,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:02,840 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:02,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:02,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:02,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:02,841 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:02,841 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@fcd3df2[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:02,841 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:02,842 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:02,846 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:02,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:02,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:02,846 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:02,846 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:02,847 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:02,847 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,847 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:02,849 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:02,849 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,849 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:02,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:02,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:02,849 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:02,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:02,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:02,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:02,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:02,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:02,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:02,853 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:02,853 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:02,853 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:02,853 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:02,853 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:02,853 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:02,855 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:02,855 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:02,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:02,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:02,855 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:02,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:02,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:02,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:02,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:02,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:02,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:02,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:02,856 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:02,857 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:02,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:02,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,861 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:02,861 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:02,861 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:02,862 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:02,862 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:02,862 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:02,862 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:02,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:02,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:02,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:02,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:02,868 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,868 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:02,868 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:02,868 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,868 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:02,868 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:02,868 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:02,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:02,874 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:02,936 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:02,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58374 deadline: 1681314902934, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,962 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:02,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:02,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:02,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:02,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:02,966 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:02,966 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:02,966 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:02,966 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:02,966 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:02,966 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:02,968 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:02,968 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:02,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:02,968 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,968 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:02,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:02,969 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:02,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:02,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:02,969 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:02,969 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:02,969 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:02,969 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:02,969 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:02,969 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:02,971 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:02,971 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:02,971 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:54:02,972 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. started... 2023-04-12 15:54:02,972 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. as already flushing 2023-04-12 15:54:02,972 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:54:02,972 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:02,972 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:02,972 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,972 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:02,972 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:02,981 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,981 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:02,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:02,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:02,982 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:02,982 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:02,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:02,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:02,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:02,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:02,984 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:02,984 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:02,984 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66657068[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:02,984 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:02,985 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:02,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:02,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:02,985 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:02,985 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:02,986 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:02,988 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,988 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:02,990 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:02,990 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:02,990 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,990 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:02,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:02,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:02,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:02,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:02,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:02,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:02,999 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:02,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58354 deadline: 1681314902997, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:03,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:03,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:03,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:03,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:03,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:03,006 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:03,006 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:03,006 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:03,006 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:03,006 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:03,006 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:03,013 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:03,013 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:03,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:03,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:03,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:03,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:03,013 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:03,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:03,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:03,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:03,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:03,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:03,014 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:03,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:03,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:03,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:03,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:03,017 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:03,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:03,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:03,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:03,020 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:03,020 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:03,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:03,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:03,021 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:03,020 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:03,021 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:03,021 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:03,020 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:03,021 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:03,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:03,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 275 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58340 deadline: 1681314903018, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:03,021 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:03,021 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:03,021 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:03,021 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:03,021 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:03,021 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:03,022 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:03,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:03,022 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:03,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58414 deadline: 1681314903021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:03,022 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:03,027 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:03,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58398 deadline: 1681314903023, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:03,122 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:03,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:03,126 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:03,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:03,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:03,128 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:03,128 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:03,128 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:03,128 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:03,128 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:03,128 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:03,130 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:03,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:03,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:03,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:03,130 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:03,130 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:03,131 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:03,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:03,131 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:03,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:03,131 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:03,133 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:03,133 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:03,133 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:03,133 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:03,133 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:03,133 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:03,133 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:03,133 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:54:03,133 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. started... 2023-04-12 15:54:03,133 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. as already flushing 2023-04-12 15:54:03,133 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:54:03,134 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:03,134 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:03,134 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:03,134 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:03,134 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:03,137 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:03,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:03,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:03,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:03,137 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:03,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:03,137 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:03,138 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:03,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:03,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:03,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:03,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:03,139 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:03,139 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e36acb2[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:03,139 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:03,139 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:03,140 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:03,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:03,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:03,140 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:03,140 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:03,141 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:03,141 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:03,141 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:03,142 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:03,142 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:03,142 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:03,142 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:03,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:03,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:03,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:03,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:03,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:03,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:03,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:03,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:03,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:03,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:03,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:03,145 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:03,145 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:03,145 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:03,145 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:03,145 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:03,145 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:03,146 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:03,146 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:03,146 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:03,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:03,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:03,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:03,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:03,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:03,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:03,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:03,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:03,147 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:03,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:03,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:03,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:03,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:03,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:03,149 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:03,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:03,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:03,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:03,155 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:03,155 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:03,155 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:03,155 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:03,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:03,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:03,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:03,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:03,155 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:03,155 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:03,155 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:03,155 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:03,155 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:03,155 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:03,155 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:03,155 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:03,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:03,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:03,240 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=743 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/C/41a203801ebb4375b408d808becc8433 2023-04-12 15:54:03,246 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/A/2366452ecfbd4c69984d3cbe534c4e92 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/2366452ecfbd4c69984d3cbe534c4e92 2023-04-12 15:54:03,251 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/2366452ecfbd4c69984d3cbe534c4e92, entries=150, sequenceid=743, filesize=11.9 K 2023-04-12 15:54:03,253 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/B/864016bf332c4f76b473a25ed5ec6977 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/864016bf332c4f76b473a25ed5ec6977 2023-04-12 15:54:03,255 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:03,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:03,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:03,261 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/864016bf332c4f76b473a25ed5ec6977, entries=100, sequenceid=743, filesize=9.5 K 2023-04-12 15:54:03,263 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/C/41a203801ebb4375b408d808becc8433 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/41a203801ebb4375b408d808becc8433 2023-04-12 15:54:03,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:03,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:03,268 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:03,268 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:03,268 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:03,268 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:03,272 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:03,272 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:03,272 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/41a203801ebb4375b408d808becc8433, entries=100, sequenceid=743, filesize=9.5 K 2023-04-12 15:54:03,273 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:03,273 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for feda8274e3aaad09f0f3a1336e663d20 in 902ms, sequenceid=743, compaction requested=true 2023-04-12 15:54:03,273 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for feda8274e3aaad09f0f3a1336e663d20: 2023-04-12 15:54:03,273 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:03,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:03,274 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:54:03,274 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-12 15:54:03,274 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-12 15:54:03,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:03,274 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:03,274 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:03,274 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:54:03,274 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:54:03,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:03,275 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:03,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:03,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:03,276 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 52032 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-12 15:54:03,276 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:03,276 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1912): feda8274e3aaad09f0f3a1336e663d20/A is initiating minor compaction (all files) 2023-04-12 15:54:03,276 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47152 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-12 15:54:03,276 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of feda8274e3aaad09f0f3a1336e663d20/A in TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:54:03,276 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1912): feda8274e3aaad09f0f3a1336e663d20/B is initiating minor compaction (all files) 2023-04-12 15:54:03,276 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/8b1c4b3960d8446cae500b3c09331f73, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/2e57c5566bfb489f9c4174afe49e367e, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/3a432ad713e8410cbe92336938680cf9, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/2366452ecfbd4c69984d3cbe534c4e92] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp, totalSize=50.8 K 2023-04-12 15:54:03,276 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2259): Starting compaction of feda8274e3aaad09f0f3a1336e663d20/B in TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:54:03,276 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:03,276 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/22b358b703e447949f97b66d09bfa35e, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/ea5502a520744fa98b5417e8e8c5954c, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/b509560954da41f28f642c7ba14b613a, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/864016bf332c4f76b473a25ed5ec6977] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp, totalSize=46.0 K 2023-04-12 15:54:03,276 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:03,277 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:03,277 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:03,277 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:03,277 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:03,277 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:03,277 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 8b1c4b3960d8446cae500b3c09331f73, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=696, earliestPutTs=1681314840124 2023-04-12 15:54:03,277 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 22b358b703e447949f97b66d09bfa35e, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=696, earliestPutTs=1681314840124 2023-04-12 15:54:03,278 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:54:03,278 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. started... 2023-04-12 15:54:03,278 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing feda8274e3aaad09f0f3a1336e663d20 3/3 column families, dataSize=134.18 KB heapSize=361.55 KB 2023-04-12 15:54:03,278 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting ea5502a520744fa98b5417e8e8c5954c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=708, earliestPutTs=1681314840235 2023-04-12 15:54:03,278 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 2e57c5566bfb489f9c4174afe49e367e, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=708, earliestPutTs=1681314840235 2023-04-12 15:54:03,278 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK feda8274e3aaad09f0f3a1336e663d20, store=A 2023-04-12 15:54:03,278 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:54:03,279 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK feda8274e3aaad09f0f3a1336e663d20, store=B 2023-04-12 15:54:03,279 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:54:03,279 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 3a432ad713e8410cbe92336938680cf9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=731, earliestPutTs=1681314840795 2023-04-12 15:54:03,279 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting b509560954da41f28f642c7ba14b613a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=731, earliestPutTs=1681314840795 2023-04-12 15:54:03,279 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK feda8274e3aaad09f0f3a1336e663d20, store=C 2023-04-12 15:54:03,279 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:54:03,279 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 2366452ecfbd4c69984d3cbe534c4e92, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=743, earliestPutTs=1681314842368 2023-04-12 15:54:03,279 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 864016bf332c4f76b473a25ed5ec6977, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=743, earliestPutTs=1681314842370 2023-04-12 15:54:03,304 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] throttle.PressureAwareThroughputController(145): feda8274e3aaad09f0f3a1336e663d20#A#compaction#88 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:54:03,305 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] throttle.PressureAwareThroughputController(145): feda8274e3aaad09f0f3a1336e663d20#B#compaction#89 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:54:03,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on feda8274e3aaad09f0f3a1336e663d20 2023-04-12 15:54:03,505 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. as already flushing 2023-04-12 15:54:03,558 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:03,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58398 deadline: 1681314903531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:03,562 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:03,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58354 deadline: 1681314903559, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:03,562 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:03,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 279 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58340 deadline: 1681314903560, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:03,563 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:03,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58414 deadline: 1681314903561, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:03,667 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:03,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58354 deadline: 1681314903663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:03,668 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:03,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 281 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58340 deadline: 1681314903663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:03,668 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:03,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58414 deadline: 1681314903664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:03,720 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/A/28ef52ffd35f4ead9054fe9bb2148d2f as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/28ef52ffd35f4ead9054fe9bb2148d2f 2023-04-12 15:54:03,729 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in feda8274e3aaad09f0f3a1336e663d20/A of feda8274e3aaad09f0f3a1336e663d20 into 28ef52ffd35f4ead9054fe9bb2148d2f(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:54:03,729 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for feda8274e3aaad09f0f3a1336e663d20: 2023-04-12 15:54:03,730 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20., storeName=feda8274e3aaad09f0f3a1336e663d20/A, priority=12, startTime=1681314843274; duration=0sec 2023-04-12 15:54:03,730 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:54:03,730 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-12 15:54:03,731 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47152 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-12 15:54:03,732 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1912): feda8274e3aaad09f0f3a1336e663d20/C is initiating minor compaction (all files) 2023-04-12 15:54:03,732 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of feda8274e3aaad09f0f3a1336e663d20/C in TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:54:03,732 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/13c7196fba4249949d37fa20ea528ea8, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/8896c8f0335045debdf87433b8682b0c, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/dd5762991a904e338ca505ec61a46c97, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/41a203801ebb4375b408d808becc8433] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp, totalSize=46.0 K 2023-04-12 15:54:03,732 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 13c7196fba4249949d37fa20ea528ea8, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=696, earliestPutTs=1681314840124 2023-04-12 15:54:03,733 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 8896c8f0335045debdf87433b8682b0c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=708, earliestPutTs=1681314840235 2023-04-12 15:54:03,733 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting dd5762991a904e338ca505ec61a46c97, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=731, earliestPutTs=1681314840795 2023-04-12 15:54:03,734 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 41a203801ebb4375b408d808becc8433, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=743, earliestPutTs=1681314842370 2023-04-12 15:54:03,742 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] throttle.PressureAwareThroughputController(145): feda8274e3aaad09f0f3a1336e663d20#C#compaction#90 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:54:03,744 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/B/b3b2f7912785464c87d84f3a48a7a73f as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/b3b2f7912785464c87d84f3a48a7a73f 2023-04-12 15:54:03,749 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in feda8274e3aaad09f0f3a1336e663d20/B of feda8274e3aaad09f0f3a1336e663d20 into b3b2f7912785464c87d84f3a48a7a73f(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:54:03,749 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for feda8274e3aaad09f0f3a1336e663d20: 2023-04-12 15:54:03,749 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20., storeName=feda8274e3aaad09f0f3a1336e663d20/B, priority=12, startTime=1681314843274; duration=0sec 2023-04-12 15:54:03,749 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:54:03,757 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=775 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/A/0524cfa42ae34064817949288100fd08 2023-04-12 15:54:03,790 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=775 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/B/c8d8da78448a4dd391a142c22443a797 2023-04-12 15:54:03,823 DEBUG [Thread-156] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4b02eeb3 to 127.0.0.1:60154 2023-04-12 15:54:03,823 DEBUG [Thread-156] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 15:54:03,823 DEBUG [Thread-154] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3cd3abbd to 127.0.0.1:60154 2023-04-12 15:54:03,823 DEBUG [Thread-154] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 15:54:03,824 DEBUG [Thread-152] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x04fb2b82 to 127.0.0.1:60154 2023-04-12 15:54:03,824 DEBUG [Thread-152] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 15:54:03,827 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=775 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/C/f850998ff6e341f6bc13c5a8918015cf 2023-04-12 15:54:03,829 DEBUG [Thread-158] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4808c6d5 to 127.0.0.1:60154 2023-04-12 15:54:03,829 DEBUG [Thread-158] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 15:54:03,834 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/A/0524cfa42ae34064817949288100fd08 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/0524cfa42ae34064817949288100fd08 2023-04-12 15:54:03,840 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/0524cfa42ae34064817949288100fd08, entries=150, sequenceid=775, filesize=11.9 K 2023-04-12 15:54:03,841 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/B/c8d8da78448a4dd391a142c22443a797 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/c8d8da78448a4dd391a142c22443a797 2023-04-12 15:54:03,847 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/c8d8da78448a4dd391a142c22443a797, entries=150, sequenceid=775, filesize=11.9 K 2023-04-12 15:54:03,848 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/C/f850998ff6e341f6bc13c5a8918015cf as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/f850998ff6e341f6bc13c5a8918015cf 2023-04-12 15:54:03,853 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/f850998ff6e341f6bc13c5a8918015cf, entries=150, sequenceid=775, filesize=11.9 K 2023-04-12 15:54:03,854 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~361.50 KB/370176, currentSize=53.67 KB/54960 for feda8274e3aaad09f0f3a1336e663d20 in 576ms, sequenceid=775, compaction requested=false 2023-04-12 15:54:03,854 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for feda8274e3aaad09f0f3a1336e663d20: 2023-04-12 15:54:03,854 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:54:03,854 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:03,854 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:03,854 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:03,854 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:03,854 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:03,856 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:03,856 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:03,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:03,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:03,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:03,856 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:03,856 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:03,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:03,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:03,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:03,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:03,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:03,857 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:03,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@249c4765[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:03,858 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:03,858 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:03,862 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:03,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:03,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:03,862 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:03,862 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:03,862 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:03,862 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:03,862 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:03,865 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:03,865 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:03,865 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:03,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:03,865 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:03,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:03,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:03,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:03,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:03,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:03,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:03,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:03,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:03,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:03,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:03,867 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:03,867 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:03,867 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:03,868 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:03,868 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:03,868 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:03,869 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:03,869 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:03,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:03,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:03,870 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:03,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:03,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:03,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:03,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:03,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:03,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:03,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:03,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:03,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:03,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:03,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:03,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:03,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:03,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:03,877 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:03,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on feda8274e3aaad09f0f3a1336e663d20 2023-04-12 15:54:03,878 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing feda8274e3aaad09f0f3a1336e663d20 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-12 15:54:03,878 DEBUG [Thread-149] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x42927198 to 127.0.0.1:60154 2023-04-12 15:54:03,878 DEBUG [Thread-149] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 15:54:03,879 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:03,882 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:03,882 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:03,882 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:03,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:03,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:03,882 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:03,882 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:03,882 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:03,882 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:03,882 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:03,882 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:03,882 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:03,883 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:03,883 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:03,883 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:03,883 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:03,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:03,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:03,887 DEBUG [Thread-145] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7e728705 to 127.0.0.1:60154 2023-04-12 15:54:03,887 DEBUG [Thread-147] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x63859c62 to 127.0.0.1:60154 2023-04-12 15:54:03,887 DEBUG [Thread-145] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 15:54:03,887 DEBUG [Thread-147] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 15:54:03,887 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK feda8274e3aaad09f0f3a1336e663d20, store=A 2023-04-12 15:54:03,888 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:03,888 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK feda8274e3aaad09f0f3a1336e663d20, store=B 2023-04-12 15:54:03,888 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:03,888 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK feda8274e3aaad09f0f3a1336e663d20, store=C 2023-04-12 15:54:03,888 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:03,983 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:03,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:04,157 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/C/af8836fb28b945898cef83c58f97a4a3 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/af8836fb28b945898cef83c58f97a4a3 2023-04-12 15:54:04,163 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in feda8274e3aaad09f0f3a1336e663d20/C of feda8274e3aaad09f0f3a1336e663d20 into af8836fb28b945898cef83c58f97a4a3(size=12.9 K), total size for store is 24.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:54:04,163 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for feda8274e3aaad09f0f3a1336e663d20: 2023-04-12 15:54:04,163 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20., storeName=feda8274e3aaad09f0f3a1336e663d20/C, priority=12, startTime=1681314843274; duration=0sec 2023-04-12 15:54:04,163 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:54:04,296 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=791 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/A/8a2aabd495bd4e5fb1b2615aae05e343 2023-04-12 15:54:04,565 DEBUG [Thread-141] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0e16d41b to 127.0.0.1:60154 2023-04-12 15:54:04,566 DEBUG [Thread-141] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 15:54:04,708 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=791 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/B/fa61b5c852ca47038510d3d57df2af5e 2023-04-12 15:54:04,951 DEBUG [Thread-143] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4fea27db to 127.0.0.1:60154 2023-04-12 15:54:04,952 DEBUG [Thread-143] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 15:54:04,952 INFO [Listener at localhost/43583] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-12 15:54:04,952 INFO [Listener at localhost/43583] hbase.AcidGuaranteesTestTool(394): wrote 114 2023-04-12 15:54:04,952 INFO [Listener at localhost/43583] hbase.AcidGuaranteesTestTool(394): wrote 119 2023-04-12 15:54:04,952 INFO [Listener at localhost/43583] hbase.AcidGuaranteesTestTool(394): wrote 159 2023-04-12 15:54:04,952 INFO [Listener at localhost/43583] hbase.AcidGuaranteesTestTool(394): wrote 171 2023-04-12 15:54:04,952 INFO [Listener at localhost/43583] hbase.AcidGuaranteesTestTool(394): wrote 133 2023-04-12 15:54:04,952 INFO [Listener at localhost/43583] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-12 15:54:04,952 INFO [Listener at localhost/43583] hbase.AcidGuaranteesTestTool(398): read 3925 2023-04-12 15:54:04,952 INFO [Listener at localhost/43583] hbase.AcidGuaranteesTestTool(398): read 3766 2023-04-12 15:54:04,952 INFO [Listener at localhost/43583] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-12 15:54:04,952 INFO [Listener at localhost/43583] hbase.AcidGuaranteesTestTool(402): scanned 1697 2023-04-12 15:54:04,952 INFO [Listener at localhost/43583] hbase.AcidGuaranteesTestTool(403): verified 5080 rows 2023-04-12 15:54:04,952 INFO [Listener at localhost/43583] hbase.AcidGuaranteesTestTool(402): scanned 1678 2023-04-12 15:54:04,952 INFO [Listener at localhost/43583] hbase.AcidGuaranteesTestTool(403): verified 5023 rows 2023-04-12 15:54:04,952 INFO [Listener at localhost/43583] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-12 15:54:04,953 DEBUG [Listener at localhost/43583] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x35594b1b to 127.0.0.1:60154 2023-04-12 15:54:04,953 DEBUG [Listener at localhost/43583] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 15:54:04,959 INFO [Listener at localhost/43583] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-12 15:54:04,963 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-12 15:54:04,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-12 15:54:04,972 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681314844972"}]},"ts":"1681314844972"} 2023-04-12 15:54:04,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-12 15:54:04,974 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-12 15:54:05,058 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-12 15:54:05,060 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=feda8274e3aaad09f0f3a1336e663d20, UNASSIGN}] 2023-04-12 15:54:05,062 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=feda8274e3aaad09f0f3a1336e663d20, UNASSIGN 2023-04-12 15:54:05,063 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=feda8274e3aaad09f0f3a1336e663d20, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:05,063 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681314845062"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681314845062"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681314845062"}]},"ts":"1681314845062"} 2023-04-12 15:54:05,064 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; CloseRegionProcedure feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739}] 2023-04-12 15:54:05,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-12 15:54:05,119 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=791 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/C/5bc8bb1c894e4ca9a802294184fca137 2023-04-12 15:54:05,124 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/A/8a2aabd495bd4e5fb1b2615aae05e343 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/8a2aabd495bd4e5fb1b2615aae05e343 2023-04-12 15:54:05,127 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/8a2aabd495bd4e5fb1b2615aae05e343, entries=150, sequenceid=791, filesize=11.9 K 2023-04-12 15:54:05,128 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/B/fa61b5c852ca47038510d3d57df2af5e as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/fa61b5c852ca47038510d3d57df2af5e 2023-04-12 15:54:05,132 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/fa61b5c852ca47038510d3d57df2af5e, entries=150, sequenceid=791, filesize=11.9 K 2023-04-12 15:54:05,133 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/C/5bc8bb1c894e4ca9a802294184fca137 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/5bc8bb1c894e4ca9a802294184fca137 2023-04-12 15:54:05,137 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/5bc8bb1c894e4ca9a802294184fca137, entries=150, sequenceid=791, filesize=11.9 K 2023-04-12 15:54:05,138 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=13.42 KB/13740 for feda8274e3aaad09f0f3a1336e663d20 in 1260ms, sequenceid=791, compaction requested=true 2023-04-12 15:54:05,138 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for feda8274e3aaad09f0f3a1336e663d20: 2023-04-12 15:54:05,139 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:54:05,139 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:54:05,139 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 15:54:05,139 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 15:54:05,139 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:54:05,140 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37546 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 15:54:05,140 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1912): feda8274e3aaad09f0f3a1336e663d20/A is initiating minor compaction (all files) 2023-04-12 15:54:05,140 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of feda8274e3aaad09f0f3a1336e663d20/A in TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:54:05,140 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37546 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 15:54:05,140 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/28ef52ffd35f4ead9054fe9bb2148d2f, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/0524cfa42ae34064817949288100fd08, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/8a2aabd495bd4e5fb1b2615aae05e343] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp, totalSize=36.7 K 2023-04-12 15:54:05,140 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1912): feda8274e3aaad09f0f3a1336e663d20/B is initiating minor compaction (all files) 2023-04-12 15:54:05,140 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2259): Starting compaction of feda8274e3aaad09f0f3a1336e663d20/B in TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:54:05,140 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/b3b2f7912785464c87d84f3a48a7a73f, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/c8d8da78448a4dd391a142c22443a797, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/fa61b5c852ca47038510d3d57df2af5e] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp, totalSize=36.7 K 2023-04-12 15:54:05,140 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 28ef52ffd35f4ead9054fe9bb2148d2f, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=743, earliestPutTs=1681314840807 2023-04-12 15:54:05,141 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting b3b2f7912785464c87d84f3a48a7a73f, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=743, earliestPutTs=1681314840807 2023-04-12 15:54:05,141 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 0524cfa42ae34064817949288100fd08, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=775, earliestPutTs=1681314842394 2023-04-12 15:54:05,141 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting c8d8da78448a4dd391a142c22443a797, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=775, earliestPutTs=1681314842394 2023-04-12 15:54:05,141 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 8a2aabd495bd4e5fb1b2615aae05e343, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=791, earliestPutTs=1681314843535 2023-04-12 15:54:05,142 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting fa61b5c852ca47038510d3d57df2af5e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=791, earliestPutTs=1681314843535 2023-04-12 15:54:05,150 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] throttle.PressureAwareThroughputController(145): feda8274e3aaad09f0f3a1336e663d20#B#compaction#96 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:54:05,156 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] throttle.PressureAwareThroughputController(145): feda8274e3aaad09f0f3a1336e663d20#A#compaction#97 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:54:05,169 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/A/9b81a78375684890b5725beb398104ae as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/9b81a78375684890b5725beb398104ae 2023-04-12 15:54:05,174 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in feda8274e3aaad09f0f3a1336e663d20/A of feda8274e3aaad09f0f3a1336e663d20 into 9b81a78375684890b5725beb398104ae(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:54:05,174 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for feda8274e3aaad09f0f3a1336e663d20: 2023-04-12 15:54:05,174 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20., storeName=feda8274e3aaad09f0f3a1336e663d20/A, priority=13, startTime=1681314845138; duration=0sec 2023-04-12 15:54:05,174 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:54:05,174 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 15:54:05,175 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37546 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 15:54:05,175 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1912): feda8274e3aaad09f0f3a1336e663d20/C is initiating minor compaction (all files) 2023-04-12 15:54:05,175 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of feda8274e3aaad09f0f3a1336e663d20/C in TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:54:05,175 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/af8836fb28b945898cef83c58f97a4a3, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/f850998ff6e341f6bc13c5a8918015cf, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/5bc8bb1c894e4ca9a802294184fca137] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp, totalSize=36.7 K 2023-04-12 15:54:05,175 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting af8836fb28b945898cef83c58f97a4a3, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=743, earliestPutTs=1681314840807 2023-04-12 15:54:05,176 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting f850998ff6e341f6bc13c5a8918015cf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=775, earliestPutTs=1681314842394 2023-04-12 15:54:05,176 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 5bc8bb1c894e4ca9a802294184fca137, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=791, earliestPutTs=1681314843535 2023-04-12 15:54:05,185 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] throttle.PressureAwareThroughputController(145): feda8274e3aaad09f0f3a1336e663d20#C#compaction#98 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:54:05,224 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close feda8274e3aaad09f0f3a1336e663d20 2023-04-12 15:54:05,224 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing feda8274e3aaad09f0f3a1336e663d20, disabling compactions & flushes 2023-04-12 15:54:05,224 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 2 compactions to complete for region TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:54:05,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-12 15:54:05,563 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/B/8535532a829046db92667660fcb76276 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/8535532a829046db92667660fcb76276 2023-04-12 15:54:05,568 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in feda8274e3aaad09f0f3a1336e663d20/B of feda8274e3aaad09f0f3a1336e663d20 into 8535532a829046db92667660fcb76276(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:54:05,568 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for feda8274e3aaad09f0f3a1336e663d20: 2023-04-12 15:54:05,568 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20., storeName=feda8274e3aaad09f0f3a1336e663d20/B, priority=13, startTime=1681314845139; duration=0sec 2023-04-12 15:54:05,569 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:54:05,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-12 15:54:05,598 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/C/2c001660e05d46a584a67c585919eaa9 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/2c001660e05d46a584a67c585919eaa9 2023-04-12 15:54:05,602 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in feda8274e3aaad09f0f3a1336e663d20/C of feda8274e3aaad09f0f3a1336e663d20 into 2c001660e05d46a584a67c585919eaa9(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:54:05,602 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for feda8274e3aaad09f0f3a1336e663d20: 2023-04-12 15:54:05,602 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:54:05,602 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20., storeName=feda8274e3aaad09f0f3a1336e663d20/C, priority=13, startTime=1681314845139; duration=0sec 2023-04-12 15:54:05,602 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:54:05,602 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. after waiting 0 ms 2023-04-12 15:54:05,602 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:54:05,602 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:54:05,602 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing feda8274e3aaad09f0f3a1336e663d20 3/3 column families, dataSize=13.42 KB heapSize=40.59 KB 2023-04-12 15:54:05,602 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK feda8274e3aaad09f0f3a1336e663d20, store=A 2023-04-12 15:54:05,603 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:05,603 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK feda8274e3aaad09f0f3a1336e663d20, store=B 2023-04-12 15:54:05,603 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:05,603 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK feda8274e3aaad09f0f3a1336e663d20, store=C 2023-04-12 15:54:05,603 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:06,009 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=800 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/A/13255613e5fc438ab770df5dbbfe9558 2023-04-12 15:54:06,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-12 15:54:06,419 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=800 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/B/deeffba2bc3b491086a2d013d1dc641f 2023-04-12 15:54:06,830 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=800 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/C/62f0b42b6ebd4777b7d6f8e1ecf28490 2023-04-12 15:54:06,834 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/A/13255613e5fc438ab770df5dbbfe9558 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/13255613e5fc438ab770df5dbbfe9558 2023-04-12 15:54:06,838 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/13255613e5fc438ab770df5dbbfe9558, entries=50, sequenceid=800, filesize=7.1 K 2023-04-12 15:54:06,839 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/B/deeffba2bc3b491086a2d013d1dc641f as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/deeffba2bc3b491086a2d013d1dc641f 2023-04-12 15:54:06,843 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/deeffba2bc3b491086a2d013d1dc641f, entries=50, sequenceid=800, filesize=7.1 K 2023-04-12 15:54:06,844 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/.tmp/C/62f0b42b6ebd4777b7d6f8e1ecf28490 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/62f0b42b6ebd4777b7d6f8e1ecf28490 2023-04-12 15:54:06,848 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/62f0b42b6ebd4777b7d6f8e1ecf28490, entries=50, sequenceid=800, filesize=7.1 K 2023-04-12 15:54:06,849 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~13.42 KB/13740, heapSize ~40.55 KB/41520, currentSize=0 B/0 for feda8274e3aaad09f0f3a1336e663d20 in 1247ms, sequenceid=800, compaction requested=false 2023-04-12 15:54:06,873 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/e029725a1c65412ebf79fbcaab14f2c7, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/32528e7e7acb46b783961424e951923d, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/025d1df469b14594a716bcec44ee239b, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/e5f68fdf2e5a4936b59f3cfbbfce051d, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/dce1631ad0724f95b7b4c1386ccfdaa9, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/6c9aeb7cb4bc41d3b501757541eb2baf, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/0be545459d50431da6044071d99f843d, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/c2497c9e947b4049aa03fa5ba5d8741e, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/c782134cb55b47929b5e8647db87f2bc, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/74dac36decc3418b95770ac6d31bdfaa, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/96e5ebbbdecd47f4a574785d8f4e4ea2, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/c2792699c7db48388370f7b2ee7057ef, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/7588c177f7c04a76abfe8543746e3c3d, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/95667d8013f0420db185fb72810d9fe8, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/98652c09fc6f47709fbb1ffcd0800b8e, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/538b0f0b3a6d478ab26459ed27598c7b, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/31b3c6bb09a8434596a142a1de704dad, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/62693f50f24345fe8e5c1672db71a03b, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/19892e6cb9534a64b07605bf9fa6deee, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/3b54c347241c4d0eb27e2ab352d1fb72, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/54cea4ee52eb4d3385730f84924733a3, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/9befdd333838463a914b86e2aa273175, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/90fbede993c741a5ae751c399cac4197, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/dd8970471d2a4331a8462a30975daad0, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/2f9b585667b141aa8d61aab9377043a9, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/8b1c4b3960d8446cae500b3c09331f73, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/2e57c5566bfb489f9c4174afe49e367e, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/3a432ad713e8410cbe92336938680cf9, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/28ef52ffd35f4ead9054fe9bb2148d2f, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/2366452ecfbd4c69984d3cbe534c4e92, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/0524cfa42ae34064817949288100fd08, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/8a2aabd495bd4e5fb1b2615aae05e343] to archive 2023-04-12 15:54:06,874 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-12 15:54:06,880 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/e029725a1c65412ebf79fbcaab14f2c7 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/e029725a1c65412ebf79fbcaab14f2c7 2023-04-12 15:54:06,882 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/32528e7e7acb46b783961424e951923d to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/32528e7e7acb46b783961424e951923d 2023-04-12 15:54:06,884 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/025d1df469b14594a716bcec44ee239b to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/025d1df469b14594a716bcec44ee239b 2023-04-12 15:54:06,885 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/e5f68fdf2e5a4936b59f3cfbbfce051d to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/e5f68fdf2e5a4936b59f3cfbbfce051d 2023-04-12 15:54:06,887 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/dce1631ad0724f95b7b4c1386ccfdaa9 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/dce1631ad0724f95b7b4c1386ccfdaa9 2023-04-12 15:54:06,888 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/6c9aeb7cb4bc41d3b501757541eb2baf to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/6c9aeb7cb4bc41d3b501757541eb2baf 2023-04-12 15:54:06,890 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/0be545459d50431da6044071d99f843d to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/0be545459d50431da6044071d99f843d 2023-04-12 15:54:06,891 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/c2497c9e947b4049aa03fa5ba5d8741e to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/c2497c9e947b4049aa03fa5ba5d8741e 2023-04-12 15:54:06,892 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/c782134cb55b47929b5e8647db87f2bc to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/c782134cb55b47929b5e8647db87f2bc 2023-04-12 15:54:06,894 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/74dac36decc3418b95770ac6d31bdfaa to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/74dac36decc3418b95770ac6d31bdfaa 2023-04-12 15:54:06,895 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/96e5ebbbdecd47f4a574785d8f4e4ea2 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/96e5ebbbdecd47f4a574785d8f4e4ea2 2023-04-12 15:54:06,896 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/c2792699c7db48388370f7b2ee7057ef to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/c2792699c7db48388370f7b2ee7057ef 2023-04-12 15:54:06,897 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/7588c177f7c04a76abfe8543746e3c3d to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/7588c177f7c04a76abfe8543746e3c3d 2023-04-12 15:54:06,898 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/95667d8013f0420db185fb72810d9fe8 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/95667d8013f0420db185fb72810d9fe8 2023-04-12 15:54:06,899 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/98652c09fc6f47709fbb1ffcd0800b8e to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/98652c09fc6f47709fbb1ffcd0800b8e 2023-04-12 15:54:06,900 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/538b0f0b3a6d478ab26459ed27598c7b to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/538b0f0b3a6d478ab26459ed27598c7b 2023-04-12 15:54:06,901 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/31b3c6bb09a8434596a142a1de704dad to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/31b3c6bb09a8434596a142a1de704dad 2023-04-12 15:54:06,903 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/62693f50f24345fe8e5c1672db71a03b to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/62693f50f24345fe8e5c1672db71a03b 2023-04-12 15:54:06,904 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/19892e6cb9534a64b07605bf9fa6deee to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/19892e6cb9534a64b07605bf9fa6deee 2023-04-12 15:54:06,905 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/3b54c347241c4d0eb27e2ab352d1fb72 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/3b54c347241c4d0eb27e2ab352d1fb72 2023-04-12 15:54:06,906 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/54cea4ee52eb4d3385730f84924733a3 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/54cea4ee52eb4d3385730f84924733a3 2023-04-12 15:54:06,907 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/9befdd333838463a914b86e2aa273175 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/9befdd333838463a914b86e2aa273175 2023-04-12 15:54:06,908 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/90fbede993c741a5ae751c399cac4197 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/90fbede993c741a5ae751c399cac4197 2023-04-12 15:54:06,910 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/dd8970471d2a4331a8462a30975daad0 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/dd8970471d2a4331a8462a30975daad0 2023-04-12 15:54:06,911 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/2f9b585667b141aa8d61aab9377043a9 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/2f9b585667b141aa8d61aab9377043a9 2023-04-12 15:54:06,912 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/8b1c4b3960d8446cae500b3c09331f73 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/8b1c4b3960d8446cae500b3c09331f73 2023-04-12 15:54:06,913 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/2e57c5566bfb489f9c4174afe49e367e to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/2e57c5566bfb489f9c4174afe49e367e 2023-04-12 15:54:06,915 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/3a432ad713e8410cbe92336938680cf9 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/3a432ad713e8410cbe92336938680cf9 2023-04-12 15:54:06,916 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/28ef52ffd35f4ead9054fe9bb2148d2f to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/28ef52ffd35f4ead9054fe9bb2148d2f 2023-04-12 15:54:06,917 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/2366452ecfbd4c69984d3cbe534c4e92 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/2366452ecfbd4c69984d3cbe534c4e92 2023-04-12 15:54:06,918 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/0524cfa42ae34064817949288100fd08 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/0524cfa42ae34064817949288100fd08 2023-04-12 15:54:06,920 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/8a2aabd495bd4e5fb1b2615aae05e343 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/8a2aabd495bd4e5fb1b2615aae05e343 2023-04-12 15:54:06,952 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/588c2dd62af14c6db4741a912bd136b4, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/f29c6b6646e74a54934655a82b02b1e9, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/29368c91015e481ea5a07516ea99be74, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/59cfd2bedaaa4e0688bb8f241c4d75d3, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/f02085ee38e44d74bf5c59e9f6f4089f, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/73fd13d4bc004b969365cce1d296fbb4, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/9ddf29945a2747a7a51787356b0ca7d9, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/7ee6b66de8594674a282cac1c1a4734e, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/8697c6e31a084364b7a5086786027a9b, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/fe8d0f5e333c407d9cbd1335556a698b, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/039c36f07b6749e0a3caef35b4b8abc0, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/03499d929f184829aefcaafcd7ef27f5, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/0664ad13f11a4e5da35911f2db3f9a28, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/44460f66245640c8835841e0145d0671, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/018e9d42612741f0ae1376b89b682c02, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/8be84eb2f3b1437a964861c66b5ee6ee, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/6dca3d7853cf4752829affa84c094274, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/6987831befee4dd7a95b462a1caab04c, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/ce27e115c43f48aba2e29720eaec8653, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/3ad20f89347e488bb98f491671452d33, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/ca8e69fdbabd46b098c4fbc93f8d95b8, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/8747c3c229ec4277881cd2644d77a4a5, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/13a6fdb6301e4ed5a5a193a58a1f0e57, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/666862bf676443dc9a35753f3b8e4389, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/22b358b703e447949f97b66d09bfa35e, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/aec10d4ea66641c69af3beef4aa280f3, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/ea5502a520744fa98b5417e8e8c5954c, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/b509560954da41f28f642c7ba14b613a, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/b3b2f7912785464c87d84f3a48a7a73f, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/864016bf332c4f76b473a25ed5ec6977, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/c8d8da78448a4dd391a142c22443a797, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/fa61b5c852ca47038510d3d57df2af5e] to archive 2023-04-12 15:54:06,953 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-12 15:54:06,955 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/588c2dd62af14c6db4741a912bd136b4 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/588c2dd62af14c6db4741a912bd136b4 2023-04-12 15:54:06,956 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/f29c6b6646e74a54934655a82b02b1e9 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/f29c6b6646e74a54934655a82b02b1e9 2023-04-12 15:54:06,957 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/29368c91015e481ea5a07516ea99be74 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/29368c91015e481ea5a07516ea99be74 2023-04-12 15:54:06,958 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/59cfd2bedaaa4e0688bb8f241c4d75d3 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/59cfd2bedaaa4e0688bb8f241c4d75d3 2023-04-12 15:54:06,960 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/f02085ee38e44d74bf5c59e9f6f4089f to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/f02085ee38e44d74bf5c59e9f6f4089f 2023-04-12 15:54:06,961 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/73fd13d4bc004b969365cce1d296fbb4 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/73fd13d4bc004b969365cce1d296fbb4 2023-04-12 15:54:06,962 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/9ddf29945a2747a7a51787356b0ca7d9 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/9ddf29945a2747a7a51787356b0ca7d9 2023-04-12 15:54:06,963 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/7ee6b66de8594674a282cac1c1a4734e to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/7ee6b66de8594674a282cac1c1a4734e 2023-04-12 15:54:06,964 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/8697c6e31a084364b7a5086786027a9b to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/8697c6e31a084364b7a5086786027a9b 2023-04-12 15:54:06,966 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/fe8d0f5e333c407d9cbd1335556a698b to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/fe8d0f5e333c407d9cbd1335556a698b 2023-04-12 15:54:06,967 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/039c36f07b6749e0a3caef35b4b8abc0 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/039c36f07b6749e0a3caef35b4b8abc0 2023-04-12 15:54:06,968 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/03499d929f184829aefcaafcd7ef27f5 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/03499d929f184829aefcaafcd7ef27f5 2023-04-12 15:54:06,969 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/0664ad13f11a4e5da35911f2db3f9a28 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/0664ad13f11a4e5da35911f2db3f9a28 2023-04-12 15:54:06,970 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/44460f66245640c8835841e0145d0671 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/44460f66245640c8835841e0145d0671 2023-04-12 15:54:06,971 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/018e9d42612741f0ae1376b89b682c02 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/018e9d42612741f0ae1376b89b682c02 2023-04-12 15:54:06,972 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/8be84eb2f3b1437a964861c66b5ee6ee to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/8be84eb2f3b1437a964861c66b5ee6ee 2023-04-12 15:54:06,973 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/6dca3d7853cf4752829affa84c094274 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/6dca3d7853cf4752829affa84c094274 2023-04-12 15:54:06,974 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/6987831befee4dd7a95b462a1caab04c to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/6987831befee4dd7a95b462a1caab04c 2023-04-12 15:54:06,975 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/ce27e115c43f48aba2e29720eaec8653 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/ce27e115c43f48aba2e29720eaec8653 2023-04-12 15:54:06,977 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/3ad20f89347e488bb98f491671452d33 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/3ad20f89347e488bb98f491671452d33 2023-04-12 15:54:06,978 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/ca8e69fdbabd46b098c4fbc93f8d95b8 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/ca8e69fdbabd46b098c4fbc93f8d95b8 2023-04-12 15:54:06,979 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/8747c3c229ec4277881cd2644d77a4a5 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/8747c3c229ec4277881cd2644d77a4a5 2023-04-12 15:54:06,980 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/13a6fdb6301e4ed5a5a193a58a1f0e57 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/13a6fdb6301e4ed5a5a193a58a1f0e57 2023-04-12 15:54:06,981 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/666862bf676443dc9a35753f3b8e4389 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/666862bf676443dc9a35753f3b8e4389 2023-04-12 15:54:06,982 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/22b358b703e447949f97b66d09bfa35e to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/22b358b703e447949f97b66d09bfa35e 2023-04-12 15:54:06,983 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/aec10d4ea66641c69af3beef4aa280f3 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/aec10d4ea66641c69af3beef4aa280f3 2023-04-12 15:54:06,984 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/ea5502a520744fa98b5417e8e8c5954c to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/ea5502a520744fa98b5417e8e8c5954c 2023-04-12 15:54:06,985 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/b509560954da41f28f642c7ba14b613a to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/b509560954da41f28f642c7ba14b613a 2023-04-12 15:54:06,987 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/b3b2f7912785464c87d84f3a48a7a73f to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/b3b2f7912785464c87d84f3a48a7a73f 2023-04-12 15:54:06,988 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/864016bf332c4f76b473a25ed5ec6977 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/864016bf332c4f76b473a25ed5ec6977 2023-04-12 15:54:06,989 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/c8d8da78448a4dd391a142c22443a797 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/c8d8da78448a4dd391a142c22443a797 2023-04-12 15:54:06,990 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/fa61b5c852ca47038510d3d57df2af5e to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/fa61b5c852ca47038510d3d57df2af5e 2023-04-12 15:54:06,994 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/c5a0d6094fee4311bf254280a3e4b08f, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/96022ac6fe824abebf8ed0dd0de3cd7c, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/648b4f8cb5934d67aa4c31ad2977e774, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/d67524d2cadf4fe2bc51050e662de6be, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/1e1c7c7b609441d083d9b948e0170c06, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/bb83c8786ab54e3fb162fac4fe55e697, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/c51630a3e68945b1bf5b2cdfd8408313, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/59ad81992fef445b846da0c27cb3b37e, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/75f0e5fa8f42449aa8707c2fb0b7ae2a, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/10feb66066be4a55bb8f1a2f2998be04, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/6312d05681f54994b07b846ccae53c44, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/cc127c4e007d44e3875a39869cef4af6, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/1eb34d0ed2534b92a56256d196c0b691, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/dc35c07fb7994813b6248e58273954ed, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/d09fb7e12b214e8e83c988672afa3a54, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/b985bf1b49d94215a9efe19d9a8e1895, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/18a512a078e9487486d16cc19306fe24, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/b297403b25a042c39756052596873237, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/d40b89f36464453ab1d40cce50cc40ec, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/5e234b651bf54320b2e8581163dbdcef, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/96f48745721b45d18a4a09e4b0cfe292, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/585d29c5d7d14cf4bf0f9e8fdef4aef5, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/3a2de69d594c4b3d9ebb6dd00f15f621, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/261fa644b7e540df93e142140c903b16, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/13c7196fba4249949d37fa20ea528ea8, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/77fc796ca1714261a869b33c6ce22f51, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/8896c8f0335045debdf87433b8682b0c, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/dd5762991a904e338ca505ec61a46c97, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/af8836fb28b945898cef83c58f97a4a3, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/41a203801ebb4375b408d808becc8433, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/f850998ff6e341f6bc13c5a8918015cf, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/5bc8bb1c894e4ca9a802294184fca137] to archive 2023-04-12 15:54:06,995 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-12 15:54:06,996 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/c5a0d6094fee4311bf254280a3e4b08f to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/c5a0d6094fee4311bf254280a3e4b08f 2023-04-12 15:54:07,005 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/96022ac6fe824abebf8ed0dd0de3cd7c to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/96022ac6fe824abebf8ed0dd0de3cd7c 2023-04-12 15:54:07,006 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/648b4f8cb5934d67aa4c31ad2977e774 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/648b4f8cb5934d67aa4c31ad2977e774 2023-04-12 15:54:07,008 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/d67524d2cadf4fe2bc51050e662de6be to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/d67524d2cadf4fe2bc51050e662de6be 2023-04-12 15:54:07,009 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/1e1c7c7b609441d083d9b948e0170c06 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/1e1c7c7b609441d083d9b948e0170c06 2023-04-12 15:54:07,010 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/bb83c8786ab54e3fb162fac4fe55e697 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/bb83c8786ab54e3fb162fac4fe55e697 2023-04-12 15:54:07,012 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/c51630a3e68945b1bf5b2cdfd8408313 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/c51630a3e68945b1bf5b2cdfd8408313 2023-04-12 15:54:07,013 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/59ad81992fef445b846da0c27cb3b37e to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/59ad81992fef445b846da0c27cb3b37e 2023-04-12 15:54:07,014 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/75f0e5fa8f42449aa8707c2fb0b7ae2a to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/75f0e5fa8f42449aa8707c2fb0b7ae2a 2023-04-12 15:54:07,016 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/10feb66066be4a55bb8f1a2f2998be04 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/10feb66066be4a55bb8f1a2f2998be04 2023-04-12 15:54:07,017 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/6312d05681f54994b07b846ccae53c44 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/6312d05681f54994b07b846ccae53c44 2023-04-12 15:54:07,019 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/cc127c4e007d44e3875a39869cef4af6 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/cc127c4e007d44e3875a39869cef4af6 2023-04-12 15:54:07,020 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/1eb34d0ed2534b92a56256d196c0b691 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/1eb34d0ed2534b92a56256d196c0b691 2023-04-12 15:54:07,021 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/dc35c07fb7994813b6248e58273954ed to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/dc35c07fb7994813b6248e58273954ed 2023-04-12 15:54:07,023 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/d09fb7e12b214e8e83c988672afa3a54 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/d09fb7e12b214e8e83c988672afa3a54 2023-04-12 15:54:07,024 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/b985bf1b49d94215a9efe19d9a8e1895 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/b985bf1b49d94215a9efe19d9a8e1895 2023-04-12 15:54:07,025 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/18a512a078e9487486d16cc19306fe24 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/18a512a078e9487486d16cc19306fe24 2023-04-12 15:54:07,026 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/b297403b25a042c39756052596873237 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/b297403b25a042c39756052596873237 2023-04-12 15:54:07,028 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/d40b89f36464453ab1d40cce50cc40ec to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/d40b89f36464453ab1d40cce50cc40ec 2023-04-12 15:54:07,029 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/5e234b651bf54320b2e8581163dbdcef to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/5e234b651bf54320b2e8581163dbdcef 2023-04-12 15:54:07,030 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/96f48745721b45d18a4a09e4b0cfe292 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/96f48745721b45d18a4a09e4b0cfe292 2023-04-12 15:54:07,031 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/585d29c5d7d14cf4bf0f9e8fdef4aef5 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/585d29c5d7d14cf4bf0f9e8fdef4aef5 2023-04-12 15:54:07,032 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/3a2de69d594c4b3d9ebb6dd00f15f621 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/3a2de69d594c4b3d9ebb6dd00f15f621 2023-04-12 15:54:07,033 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/261fa644b7e540df93e142140c903b16 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/261fa644b7e540df93e142140c903b16 2023-04-12 15:54:07,034 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/13c7196fba4249949d37fa20ea528ea8 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/13c7196fba4249949d37fa20ea528ea8 2023-04-12 15:54:07,035 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/77fc796ca1714261a869b33c6ce22f51 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/77fc796ca1714261a869b33c6ce22f51 2023-04-12 15:54:07,036 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/8896c8f0335045debdf87433b8682b0c to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/8896c8f0335045debdf87433b8682b0c 2023-04-12 15:54:07,037 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/dd5762991a904e338ca505ec61a46c97 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/dd5762991a904e338ca505ec61a46c97 2023-04-12 15:54:07,038 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/af8836fb28b945898cef83c58f97a4a3 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/af8836fb28b945898cef83c58f97a4a3 2023-04-12 15:54:07,039 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/41a203801ebb4375b408d808becc8433 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/41a203801ebb4375b408d808becc8433 2023-04-12 15:54:07,040 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/f850998ff6e341f6bc13c5a8918015cf to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/f850998ff6e341f6bc13c5a8918015cf 2023-04-12 15:54:07,041 DEBUG [StoreCloser-TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/5bc8bb1c894e4ca9a802294184fca137 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/5bc8bb1c894e4ca9a802294184fca137 2023-04-12 15:54:07,045 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/recovered.edits/803.seqid, newMaxSeqId=803, maxSeqId=1 2023-04-12 15:54:07,046 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20. 2023-04-12 15:54:07,047 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for feda8274e3aaad09f0f3a1336e663d20: 2023-04-12 15:54:07,048 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed feda8274e3aaad09f0f3a1336e663d20 2023-04-12 15:54:07,049 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=feda8274e3aaad09f0f3a1336e663d20, regionState=CLOSED 2023-04-12 15:54:07,049 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681314847049"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681314847049"}]},"ts":"1681314847049"} 2023-04-12 15:54:07,052 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-04-12 15:54:07,052 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; CloseRegionProcedure feda8274e3aaad09f0f3a1336e663d20, server=jenkins-hbase4.apache.org,38905,1681314818739 in 1.9860 sec 2023-04-12 15:54:07,053 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-04-12 15:54:07,053 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=feda8274e3aaad09f0f3a1336e663d20, UNASSIGN in 1.9920 sec 2023-04-12 15:54:07,054 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681314847054"}]},"ts":"1681314847054"} 2023-04-12 15:54:07,055 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-12 15:54:07,056 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-12 15:54:07,058 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 2.0930 sec 2023-04-12 15:54:07,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-12 15:54:07,078 INFO [Listener at localhost/43583] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 12 completed 2023-04-12 15:54:07,082 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-12 15:54:07,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure2.ProcedureExecutor(1029): Stored pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-12 15:54:07,089 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-12 15:54:07,091 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=15, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-12 15:54:07,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-12 15:54:07,095 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20 2023-04-12 15:54:07,098 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A, FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B, FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C, FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/recovered.edits] 2023-04-12 15:54:07,102 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/13255613e5fc438ab770df5dbbfe9558 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/13255613e5fc438ab770df5dbbfe9558 2023-04-12 15:54:07,103 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/9b81a78375684890b5725beb398104ae to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/A/9b81a78375684890b5725beb398104ae 2023-04-12 15:54:07,107 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/8535532a829046db92667660fcb76276 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/8535532a829046db92667660fcb76276 2023-04-12 15:54:07,109 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/deeffba2bc3b491086a2d013d1dc641f to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/B/deeffba2bc3b491086a2d013d1dc641f 2023-04-12 15:54:07,111 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/2c001660e05d46a584a67c585919eaa9 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/2c001660e05d46a584a67c585919eaa9 2023-04-12 15:54:07,112 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/62f0b42b6ebd4777b7d6f8e1ecf28490 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/C/62f0b42b6ebd4777b7d6f8e1ecf28490 2023-04-12 15:54:07,115 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/recovered.edits/803.seqid to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20/recovered.edits/803.seqid 2023-04-12 15:54:07,116 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/feda8274e3aaad09f0f3a1336e663d20 2023-04-12 15:54:07,116 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-12 15:54:07,118 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=15, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-12 15:54:07,139 WARN [PEWorker-4] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-12 15:54:07,188 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-12 15:54:07,189 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=15, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-12 15:54:07,189 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-12 15:54:07,189 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681314847189"}]},"ts":"9223372036854775807"} 2023-04-12 15:54:07,192 INFO [PEWorker-4] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-12 15:54:07,192 DEBUG [PEWorker-4] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => feda8274e3aaad09f0f3a1336e663d20, NAME => 'TestAcidGuarantees,,1681314822571.feda8274e3aaad09f0f3a1336e663d20.', STARTKEY => '', ENDKEY => ''}] 2023-04-12 15:54:07,192 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-12 15:54:07,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-12 15:54:07,192 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681314847192"}]},"ts":"9223372036854775807"} 2023-04-12 15:54:07,194 INFO [PEWorker-4] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-12 15:54:07,196 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(130): Finished pid=15, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-12 15:54:07,198 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=15, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 113 msec 2023-04-12 15:54:07,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-12 15:54:07,393 INFO [Listener at localhost/43583] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 15 completed 2023-04-12 15:54:07,410 INFO [Listener at localhost/43583] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=238 (was 207) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1704733776_15 at /127.0.0.1:40804 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:0;jenkins-hbase4:38905-shortCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-7 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-9 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1704733776_15 at /127.0.0.1:41022 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RSProcedureDispatcher-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1704733776_15 at /127.0.0.1:40920 [Waiting for operation #7] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x5279ba2c-shared-pool-20 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1704733776_15 at /127.0.0.1:40880 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-12 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x5279ba2c-shared-pool-21 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-8 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x5279ba2c-shared-pool-18 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-13 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-1-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1704733776_15 at /127.0.0.1:40822 [Waiting for operation #5] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x5279ba2c-shared-pool-19 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-11 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-10 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: (jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=578 (was 470) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=459 (was 354) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=672 (was 1263) 2023-04-12 15:54:07,421 INFO [Listener at localhost/43583] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=238, OpenFileDescriptor=578, MaxFileDescriptor=60000, SystemLoadAverage=459, ProcessCount=169, AvailableMemoryMB=671 2023-04-12 15:54:07,423 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-12 15:54:07,424 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-12 15:54:07,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure2.ProcedureExecutor(1029): Stored pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-12 15:54:07,426 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-12 15:54:07,426 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 16 2023-04-12 15:54:07,427 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-12 15:54:07,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-12 15:54:07,428 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8 2023-04-12 15:54:07,429 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8 empty. 2023-04-12 15:54:07,429 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8 2023-04-12 15:54:07,429 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-12 15:54:07,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-12 15:54:07,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-12 15:54:07,838 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-12 15:54:07,840 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => ca92eda5f752c4c8f5e2f01f66bc73d8, NAME => 'TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp 2023-04-12 15:54:07,846 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-12 15:54:07,846 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing ca92eda5f752c4c8f5e2f01f66bc73d8, disabling compactions & flushes 2023-04-12 15:54:07,846 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:07,846 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:07,846 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. after waiting 0 ms 2023-04-12 15:54:07,846 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:07,846 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:07,846 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for ca92eda5f752c4c8f5e2f01f66bc73d8: 2023-04-12 15:54:07,848 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-12 15:54:07,849 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681314847849"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681314847849"}]},"ts":"1681314847849"} 2023-04-12 15:54:07,850 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-12 15:54:07,851 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-12 15:54:07,851 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681314847851"}]},"ts":"1681314847851"} 2023-04-12 15:54:07,852 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-12 15:54:07,856 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=ca92eda5f752c4c8f5e2f01f66bc73d8, ASSIGN}] 2023-04-12 15:54:07,857 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=ca92eda5f752c4c8f5e2f01f66bc73d8, ASSIGN 2023-04-12 15:54:07,858 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=ca92eda5f752c4c8f5e2f01f66bc73d8, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,38905,1681314818739; forceNewPlan=false, retain=false 2023-04-12 15:54:08,009 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=ca92eda5f752c4c8f5e2f01f66bc73d8, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:08,009 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681314848009"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681314848009"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681314848009"}]},"ts":"1681314848009"} 2023-04-12 15:54:08,011 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=17, state=RUNNABLE; OpenRegionProcedure ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739}] 2023-04-12 15:54:08,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-12 15:54:08,168 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:08,169 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => ca92eda5f752c4c8f5e2f01f66bc73d8, NAME => 'TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.', STARTKEY => '', ENDKEY => ''} 2023-04-12 15:54:08,169 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees ca92eda5f752c4c8f5e2f01f66bc73d8 2023-04-12 15:54:08,169 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-12 15:54:08,169 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for ca92eda5f752c4c8f5e2f01f66bc73d8 2023-04-12 15:54:08,169 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for ca92eda5f752c4c8f5e2f01f66bc73d8 2023-04-12 15:54:08,173 INFO [StoreOpener-ca92eda5f752c4c8f5e2f01f66bc73d8-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region ca92eda5f752c4c8f5e2f01f66bc73d8 2023-04-12 15:54:08,175 DEBUG [StoreOpener-ca92eda5f752c4c8f5e2f01f66bc73d8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A 2023-04-12 15:54:08,175 DEBUG [StoreOpener-ca92eda5f752c4c8f5e2f01f66bc73d8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A 2023-04-12 15:54:08,175 INFO [StoreOpener-ca92eda5f752c4c8f5e2f01f66bc73d8-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-12 15:54:08,175 INFO [StoreOpener-ca92eda5f752c4c8f5e2f01f66bc73d8-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ca92eda5f752c4c8f5e2f01f66bc73d8 columnFamilyName A 2023-04-12 15:54:08,176 INFO [StoreOpener-ca92eda5f752c4c8f5e2f01f66bc73d8-1] regionserver.HStore(310): Store=ca92eda5f752c4c8f5e2f01f66bc73d8/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 15:54:08,176 INFO [StoreOpener-ca92eda5f752c4c8f5e2f01f66bc73d8-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region ca92eda5f752c4c8f5e2f01f66bc73d8 2023-04-12 15:54:08,178 DEBUG [StoreOpener-ca92eda5f752c4c8f5e2f01f66bc73d8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B 2023-04-12 15:54:08,178 DEBUG [StoreOpener-ca92eda5f752c4c8f5e2f01f66bc73d8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B 2023-04-12 15:54:08,178 INFO [StoreOpener-ca92eda5f752c4c8f5e2f01f66bc73d8-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-12 15:54:08,178 INFO [StoreOpener-ca92eda5f752c4c8f5e2f01f66bc73d8-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ca92eda5f752c4c8f5e2f01f66bc73d8 columnFamilyName B 2023-04-12 15:54:08,179 INFO [StoreOpener-ca92eda5f752c4c8f5e2f01f66bc73d8-1] regionserver.HStore(310): Store=ca92eda5f752c4c8f5e2f01f66bc73d8/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 15:54:08,179 INFO [StoreOpener-ca92eda5f752c4c8f5e2f01f66bc73d8-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region ca92eda5f752c4c8f5e2f01f66bc73d8 2023-04-12 15:54:08,180 DEBUG [StoreOpener-ca92eda5f752c4c8f5e2f01f66bc73d8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C 2023-04-12 15:54:08,180 DEBUG [StoreOpener-ca92eda5f752c4c8f5e2f01f66bc73d8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C 2023-04-12 15:54:08,181 INFO [StoreOpener-ca92eda5f752c4c8f5e2f01f66bc73d8-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-12 15:54:08,181 INFO [StoreOpener-ca92eda5f752c4c8f5e2f01f66bc73d8-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ca92eda5f752c4c8f5e2f01f66bc73d8 columnFamilyName C 2023-04-12 15:54:08,181 INFO [StoreOpener-ca92eda5f752c4c8f5e2f01f66bc73d8-1] regionserver.HStore(310): Store=ca92eda5f752c4c8f5e2f01f66bc73d8/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 15:54:08,182 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:08,182 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8 2023-04-12 15:54:08,183 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8 2023-04-12 15:54:08,185 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-12 15:54:08,187 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for ca92eda5f752c4c8f5e2f01f66bc73d8 2023-04-12 15:54:08,188 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-12 15:54:08,189 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened ca92eda5f752c4c8f5e2f01f66bc73d8; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64829276, jitterRate=-0.03396850824356079}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-12 15:54:08,189 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for ca92eda5f752c4c8f5e2f01f66bc73d8: 2023-04-12 15:54:08,190 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8., pid=18, masterSystemTime=1681314848162 2023-04-12 15:54:08,191 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:08,191 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:08,192 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=ca92eda5f752c4c8f5e2f01f66bc73d8, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:08,192 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681314848192"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681314848192"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681314848192"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681314848192"}]},"ts":"1681314848192"} 2023-04-12 15:54:08,195 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=17 2023-04-12 15:54:08,195 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=17, state=SUCCESS; OpenRegionProcedure ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 in 183 msec 2023-04-12 15:54:08,197 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=16 2023-04-12 15:54:08,197 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=16, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=ca92eda5f752c4c8f5e2f01f66bc73d8, ASSIGN in 339 msec 2023-04-12 15:54:08,198 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-12 15:54:08,198 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681314848198"}]},"ts":"1681314848198"} 2023-04-12 15:54:08,199 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-12 15:54:08,201 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-12 15:54:08,203 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=16, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 777 msec 2023-04-12 15:54:08,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-12 15:54:08,531 INFO [Listener at localhost/43583] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 16 completed 2023-04-12 15:54:08,533 DEBUG [Listener at localhost/43583] zookeeper.ReadOnlyZKClient(139): Connect 0x5b4fcfe0 to 127.0.0.1:60154 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 15:54:08,537 DEBUG [Listener at localhost/43583] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@71ea1f5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 15:54:08,539 DEBUG [Listener at localhost/43583] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 15:54:08,543 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34062, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 15:54:08,544 DEBUG [Listener at localhost/43583] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-12 15:54:08,547 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37742, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-12 15:54:08,554 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-12 15:54:08,554 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-12 15:54:08,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure2.ProcedureExecutor(1029): Stored pid=19, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-12 15:54:08,586 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-12 15:54:08,587 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-12 15:54:08,587 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-12 15:54:08,592 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=19, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-12 15:54:08,609 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=ca92eda5f752c4c8f5e2f01f66bc73d8, REOPEN/MOVE}] 2023-04-12 15:54:08,610 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=ca92eda5f752c4c8f5e2f01f66bc73d8, REOPEN/MOVE 2023-04-12 15:54:08,611 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=ca92eda5f752c4c8f5e2f01f66bc73d8, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:08,611 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681314848611"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681314848611"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681314848611"}]},"ts":"1681314848611"} 2023-04-12 15:54:08,613 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=21, state=RUNNABLE; CloseRegionProcedure ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739}] 2023-04-12 15:54:08,765 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close ca92eda5f752c4c8f5e2f01f66bc73d8 2023-04-12 15:54:08,766 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing ca92eda5f752c4c8f5e2f01f66bc73d8, disabling compactions & flushes 2023-04-12 15:54:08,766 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:08,766 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:08,766 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. after waiting 0 ms 2023-04-12 15:54:08,766 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:08,770 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-12 15:54:08,771 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:08,771 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for ca92eda5f752c4c8f5e2f01f66bc73d8: 2023-04-12 15:54:08,771 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: ca92eda5f752c4c8f5e2f01f66bc73d8 to self. 2023-04-12 15:54:08,773 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed ca92eda5f752c4c8f5e2f01f66bc73d8 2023-04-12 15:54:08,773 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=ca92eda5f752c4c8f5e2f01f66bc73d8, regionState=CLOSED 2023-04-12 15:54:08,773 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681314848773"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681314848773"}]},"ts":"1681314848773"} 2023-04-12 15:54:08,776 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=21 2023-04-12 15:54:08,777 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=21, state=SUCCESS; CloseRegionProcedure ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 in 163 msec 2023-04-12 15:54:08,777 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=ca92eda5f752c4c8f5e2f01f66bc73d8, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,38905,1681314818739; forceNewPlan=false, retain=true 2023-04-12 15:54:08,928 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=ca92eda5f752c4c8f5e2f01f66bc73d8, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:08,928 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681314848928"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681314848928"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681314848928"}]},"ts":"1681314848928"} 2023-04-12 15:54:08,930 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=21, state=RUNNABLE; OpenRegionProcedure ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739}] 2023-04-12 15:54:09,085 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:09,085 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => ca92eda5f752c4c8f5e2f01f66bc73d8, NAME => 'TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.', STARTKEY => '', ENDKEY => ''} 2023-04-12 15:54:09,086 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees ca92eda5f752c4c8f5e2f01f66bc73d8 2023-04-12 15:54:09,086 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-12 15:54:09,086 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for ca92eda5f752c4c8f5e2f01f66bc73d8 2023-04-12 15:54:09,086 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for ca92eda5f752c4c8f5e2f01f66bc73d8 2023-04-12 15:54:09,088 INFO [StoreOpener-ca92eda5f752c4c8f5e2f01f66bc73d8-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region ca92eda5f752c4c8f5e2f01f66bc73d8 2023-04-12 15:54:09,089 DEBUG [StoreOpener-ca92eda5f752c4c8f5e2f01f66bc73d8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A 2023-04-12 15:54:09,089 DEBUG [StoreOpener-ca92eda5f752c4c8f5e2f01f66bc73d8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A 2023-04-12 15:54:09,090 INFO [StoreOpener-ca92eda5f752c4c8f5e2f01f66bc73d8-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-12 15:54:09,092 INFO [StoreOpener-ca92eda5f752c4c8f5e2f01f66bc73d8-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ca92eda5f752c4c8f5e2f01f66bc73d8 columnFamilyName A 2023-04-12 15:54:09,094 INFO [StoreOpener-ca92eda5f752c4c8f5e2f01f66bc73d8-1] regionserver.HStore(310): Store=ca92eda5f752c4c8f5e2f01f66bc73d8/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 15:54:09,094 INFO [StoreOpener-ca92eda5f752c4c8f5e2f01f66bc73d8-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region ca92eda5f752c4c8f5e2f01f66bc73d8 2023-04-12 15:54:09,095 DEBUG [StoreOpener-ca92eda5f752c4c8f5e2f01f66bc73d8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B 2023-04-12 15:54:09,095 DEBUG [StoreOpener-ca92eda5f752c4c8f5e2f01f66bc73d8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B 2023-04-12 15:54:09,095 INFO [StoreOpener-ca92eda5f752c4c8f5e2f01f66bc73d8-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-12 15:54:09,096 INFO [StoreOpener-ca92eda5f752c4c8f5e2f01f66bc73d8-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ca92eda5f752c4c8f5e2f01f66bc73d8 columnFamilyName B 2023-04-12 15:54:09,096 INFO [StoreOpener-ca92eda5f752c4c8f5e2f01f66bc73d8-1] regionserver.HStore(310): Store=ca92eda5f752c4c8f5e2f01f66bc73d8/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 15:54:09,096 INFO [StoreOpener-ca92eda5f752c4c8f5e2f01f66bc73d8-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region ca92eda5f752c4c8f5e2f01f66bc73d8 2023-04-12 15:54:09,097 DEBUG [StoreOpener-ca92eda5f752c4c8f5e2f01f66bc73d8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C 2023-04-12 15:54:09,097 DEBUG [StoreOpener-ca92eda5f752c4c8f5e2f01f66bc73d8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C 2023-04-12 15:54:09,097 INFO [StoreOpener-ca92eda5f752c4c8f5e2f01f66bc73d8-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-12 15:54:09,098 INFO [StoreOpener-ca92eda5f752c4c8f5e2f01f66bc73d8-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ca92eda5f752c4c8f5e2f01f66bc73d8 columnFamilyName C 2023-04-12 15:54:09,098 INFO [StoreOpener-ca92eda5f752c4c8f5e2f01f66bc73d8-1] regionserver.HStore(310): Store=ca92eda5f752c4c8f5e2f01f66bc73d8/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 15:54:09,099 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:09,099 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8 2023-04-12 15:54:09,100 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8 2023-04-12 15:54:09,102 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-12 15:54:09,103 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for ca92eda5f752c4c8f5e2f01f66bc73d8 2023-04-12 15:54:09,104 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened ca92eda5f752c4c8f5e2f01f66bc73d8; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64902956, jitterRate=-0.032870590686798096}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-12 15:54:09,104 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for ca92eda5f752c4c8f5e2f01f66bc73d8: 2023-04-12 15:54:09,104 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8., pid=23, masterSystemTime=1681314849081 2023-04-12 15:54:09,106 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:09,106 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:09,106 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=ca92eda5f752c4c8f5e2f01f66bc73d8, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,106 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681314849106"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681314849106"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681314849106"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681314849106"}]},"ts":"1681314849106"} 2023-04-12 15:54:09,109 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=21 2023-04-12 15:54:09,109 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=21, state=SUCCESS; OpenRegionProcedure ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 in 177 msec 2023-04-12 15:54:09,110 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=20 2023-04-12 15:54:09,110 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=20, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=ca92eda5f752c4c8f5e2f01f66bc73d8, REOPEN/MOVE in 500 msec 2023-04-12 15:54:09,112 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=19 2023-04-12 15:54:09,112 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=19, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 520 msec 2023-04-12 15:54:09,114 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=19, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 556 msec 2023-04-12 15:54:09,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1227): Checking to see if procedure is done pid=19 2023-04-12 15:54:09,123 DEBUG [Listener at localhost/43583] zookeeper.ReadOnlyZKClient(139): Connect 0x0a71edf0 to 127.0.0.1:60154 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 15:54:09,128 DEBUG [Listener at localhost/43583] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1441c9f0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 15:54:09,128 DEBUG [Listener at localhost/43583] zookeeper.ReadOnlyZKClient(139): Connect 0x7aba8e46 to 127.0.0.1:60154 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 15:54:09,134 DEBUG [Listener at localhost/43583] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@370947cb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 15:54:09,134 DEBUG [Listener at localhost/43583] zookeeper.ReadOnlyZKClient(139): Connect 0x0feb770d to 127.0.0.1:60154 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 15:54:09,138 DEBUG [Listener at localhost/43583] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@210d1008, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 15:54:09,139 DEBUG [Listener at localhost/43583] zookeeper.ReadOnlyZKClient(139): Connect 0x00c2598b to 127.0.0.1:60154 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 15:54:09,144 DEBUG [Listener at localhost/43583] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@498694cb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 15:54:09,144 DEBUG [Listener at localhost/43583] zookeeper.ReadOnlyZKClient(139): Connect 0x508246cc to 127.0.0.1:60154 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 15:54:09,148 DEBUG [Listener at localhost/43583] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@421e5356, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 15:54:09,148 DEBUG [Listener at localhost/43583] zookeeper.ReadOnlyZKClient(139): Connect 0x4a0fd7b4 to 127.0.0.1:60154 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 15:54:09,156 DEBUG [Listener at localhost/43583] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7a3bb940, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 15:54:09,157 DEBUG [Listener at localhost/43583] zookeeper.ReadOnlyZKClient(139): Connect 0x1a7b66a3 to 127.0.0.1:60154 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 15:54:09,160 DEBUG [Listener at localhost/43583] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2efdb02f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 15:54:09,160 DEBUG [Listener at localhost/43583] zookeeper.ReadOnlyZKClient(139): Connect 0x42d325f4 to 127.0.0.1:60154 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 15:54:09,164 DEBUG [Listener at localhost/43583] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3dc1d73, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 15:54:09,164 DEBUG [Listener at localhost/43583] zookeeper.ReadOnlyZKClient(139): Connect 0x1676475f to 127.0.0.1:60154 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 15:54:09,167 DEBUG [Listener at localhost/43583] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4d512ce0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 15:54:09,171 DEBUG [hconnection-0x6271c7c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 15:54:09,171 DEBUG [hconnection-0x62ab0eef-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 15:54:09,172 DEBUG [hconnection-0x4d1e76c7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 15:54:09,172 DEBUG [hconnection-0x73dc9b3d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 15:54:09,172 DEBUG [hconnection-0x75663512-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 15:54:09,173 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34076, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 15:54:09,174 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34094, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 15:54:09,183 DEBUG [hconnection-0x5f106170-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 15:54:09,185 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34066, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 15:54:09,185 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34078, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 15:54:09,187 DEBUG [hconnection-0x585e19c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 15:54:09,187 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34118, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 15:54:09,188 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34122, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 15:54:09,190 DEBUG [hconnection-0x36f44e49-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 15:54:09,193 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34102, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 15:54:09,197 DEBUG [hconnection-0x2e3d65b1-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 15:54:09,209 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34134, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 15:54:09,219 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:09,225 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34142, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 15:54:09,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on ca92eda5f752c4c8f5e2f01f66bc73d8 2023-04-12 15:54:09,228 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ca92eda5f752c4c8f5e2f01f66bc73d8 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-12 15:54:09,228 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca92eda5f752c4c8f5e2f01f66bc73d8, store=A 2023-04-12 15:54:09,229 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:09,229 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca92eda5f752c4c8f5e2f01f66bc73d8, store=B 2023-04-12 15:54:09,229 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:09,229 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca92eda5f752c4c8f5e2f01f66bc73d8, store=C 2023-04-12 15:54:09,229 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:09,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:09,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:09,241 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:09,241 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:09,241 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:09,241 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:09,263 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681314849248, max timestamp=1681314849260Num uniques 859; 2023-04-12 15:54:09,263 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681314849248, max timestamp=1681314849260Num uniques 859; 2023-04-12 15:54:09,263 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-12 15:54:09,263 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681314849248, max timestamp=1681314849260Num uniques 859; 2023-04-12 15:54:09,263 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-12 15:54:09,263 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-12 15:54:09,264 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:09,264 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:09,266 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:09,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:09,266 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:09,267 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,267 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:09,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:09,268 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:09,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:09,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:09,268 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:09,277 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:09,277 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:09,277 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:09,277 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:09,277 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:09,285 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:09,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34094 deadline: 1681314909279, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,285 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:09,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34066 deadline: 1681314909279, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,285 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:09,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34118 deadline: 1681314909280, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,286 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:09,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34134 deadline: 1681314909282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,287 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:09,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34122 deadline: 1681314909285, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,290 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:09,291 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:09,291 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:09,291 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. started... 2023-04-12 15:54:09,291 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. as already flushing 2023-04-12 15:54:09,291 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:09,291 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:09,291 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:09,291 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,291 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:09,291 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:09,293 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:09,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:09,293 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:09,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:09,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:09,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:09,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:09,296 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:09,296 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:09,296 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:09,296 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4258cca9[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:09,296 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:09,296 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:09,297 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:09,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:09,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:09,297 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:09,297 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:09,297 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:09,297 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,298 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:09,300 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:09,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:09,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:09,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:09,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:09,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:09,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:09,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,304 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:09,304 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:09,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:09,304 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:09,304 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:09,305 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:09,305 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:09,305 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:09,305 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:09,304 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:09,307 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:09,308 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:09,308 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:09,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:09,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:09,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:09,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:09,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:09,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:09,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:09,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:09,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:09,309 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:09,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:09,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:09,312 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:09,312 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230412c15e05e4381a4cb99d7331f030b351ea to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412c15e05e4381a4cb99d7331f030b351ea 2023-04-12 15:54:09,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:09,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,315 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:09,315 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:09,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:09,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:09,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:09,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:09,316 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:09,316 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:09,316 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:09,316 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,316 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:09,316 DEBUG [Thread-831] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:09,316 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:09,316 DEBUG [Thread-831] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:09,316 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:09,316 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,316 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:09,316 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:09,316 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:09,387 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:09,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34094 deadline: 1681314909387, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,387 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:09,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34066 deadline: 1681314909387, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,391 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:09,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34118 deadline: 1681314909388, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,391 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:09,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34134 deadline: 1681314909388, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,392 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:09,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34122 deadline: 1681314909388, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,416 DEBUG [Thread-831] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:09,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:09,420 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:09,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:09,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:09,444 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:09,444 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:09,444 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:09,444 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:09,445 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:09,445 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:09,446 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:09,446 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:09,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:09,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:09,447 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,447 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:09,447 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:09,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:09,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:09,448 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:09,452 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:09,452 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:09,452 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:09,452 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:09,452 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:09,456 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:09,456 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:09,456 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:09,456 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. started... 2023-04-12 15:54:09,456 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. as already flushing 2023-04-12 15:54:09,456 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:09,456 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:09,456 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:09,456 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,456 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:09,456 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:09,458 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,458 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:09,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:09,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:09,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:09,459 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:09,459 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:09,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:09,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:09,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:09,461 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:09,461 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:09,461 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f7302ba[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:09,461 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:09,465 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:09,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:09,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:09,465 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:09,465 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:09,465 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:09,466 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,466 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:09,469 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:09,469 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,469 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:09,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,469 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:09,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:09,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:09,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:09,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:09,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:09,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:09,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:09,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:09,472 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:09,472 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:09,472 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:09,472 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:09,472 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:09,472 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:09,473 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:09,473 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:09,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:09,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:09,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:09,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:09,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:09,473 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:09,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:09,474 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:09,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:09,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:09,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:09,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:09,476 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:09,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:09,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:09,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,479 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,479 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:09,480 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:09,480 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:09,480 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:09,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:09,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:09,480 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:09,480 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,480 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:09,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:09,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:09,480 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:09,480 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:09,480 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:09,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:09,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:09,481 DEBUG [Thread-831] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:09,481 DEBUG [Thread-831] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:09,581 DEBUG [Thread-831] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:09,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:09,586 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:09,592 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:09,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34094 deadline: 1681314909589, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,593 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:09,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34066 deadline: 1681314909592, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,595 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:09,595 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:09,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34118 deadline: 1681314909593, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34134 deadline: 1681314909593, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:09,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:09,596 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:09,596 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:09,596 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:09,596 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:09,597 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:09,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34122 deadline: 1681314909594, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,597 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:09,597 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:09,598 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,598 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:09,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:09,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:09,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:09,599 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:09,599 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,599 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:09,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:09,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:09,600 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:09,600 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:09,600 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:09,600 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:09,600 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:09,600 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:09,604 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:09,604 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:09,604 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:09,604 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. started... 2023-04-12 15:54:09,604 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. as already flushing 2023-04-12 15:54:09,604 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:09,604 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:09,604 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:09,604 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,608 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:09,608 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:09,610 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,610 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:09,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:09,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:09,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:09,611 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:09,611 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:09,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:09,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:09,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:09,612 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:09,612 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@547b6cda[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:09,612 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:09,612 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:09,615 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:09,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:09,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:09,615 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,615 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:09,615 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:09,615 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:09,615 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:09,618 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,618 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:09,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:09,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:09,618 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:09,618 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:09,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:09,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:09,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:09,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:09,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:09,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:09,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:09,622 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:09,622 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:09,622 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:09,622 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:09,622 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:09,624 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:09,624 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:09,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:09,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:09,624 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:09,625 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,626 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,628 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,628 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:09,628 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:09,628 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:09,629 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:09,628 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:09,629 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,629 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:09,629 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:09,629 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:09,629 DEBUG [Thread-831] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:09,629 DEBUG [Thread-831] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:09,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:09,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:09,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:09,632 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:09,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:09,632 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:09,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:09,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:09,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:09,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:09,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:09,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:09,633 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:09,724 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=14, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/A/241794289536461ab44241ae2b5cd066 2023-04-12 15:54:09,729 DEBUG [Thread-831] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:09,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:09,736 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:09,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:09,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:09,741 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:09,741 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:09,741 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:09,741 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:09,742 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:09,742 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:09,743 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:09,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:09,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:09,744 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:09,745 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,745 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:09,745 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:09,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:09,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:09,745 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:09,760 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:09,760 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:09,760 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:09,760 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:09,760 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:09,768 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:09,768 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:09,768 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:09,768 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. started... 2023-04-12 15:54:09,769 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. as already flushing 2023-04-12 15:54:09,772 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:09,772 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:09,772 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:09,772 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,772 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:09,772 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:09,775 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:09,775 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:09,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:09,777 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:09,777 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:09,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:09,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:09,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:09,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:09,778 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:09,778 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@121aae5f[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:09,778 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:09,778 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:09,780 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:09,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:09,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:09,780 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:09,780 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:09,781 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:09,780 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,781 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:09,782 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,782 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:09,782 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:09,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:09,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:09,782 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:09,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:09,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:09,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:09,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:09,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:09,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:09,785 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:09,785 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:09,785 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:09,786 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:09,786 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:09,786 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:09,788 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:09,788 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:09,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:09,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:09,788 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:09,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:09,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:09,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:09,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:09,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:09,789 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:09,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:09,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:09,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:09,791 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:09,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:09,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:09,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,795 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,795 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:09,795 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:09,795 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,795 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:09,795 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:09,795 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:09,795 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:09,795 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:09,795 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:09,795 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:09,796 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:09,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:09,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:09,796 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:09,796 DEBUG [Thread-831] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:09,796 DEBUG [Thread-831] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:09,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:09,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:09,807 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/B/890822e2f7634ea9b23c646406f620ce 2023-04-12 15:54:09,896 DEBUG [Thread-831] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:09,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:09,899 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:09,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34066 deadline: 1681314909898, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,899 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:09,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34094 deadline: 1681314909898, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,903 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:09,904 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:09,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34122 deadline: 1681314909903, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,905 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:09,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34134 deadline: 1681314909904, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:09,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:09,914 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:09,914 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:09,914 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:09,914 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:09,914 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:09,915 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:09,916 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:09,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34118 deadline: 1681314909902, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,916 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,917 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,917 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:09,917 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:09,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:09,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:09,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:09,918 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:09,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:09,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:09,918 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:09,933 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:09,933 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:09,933 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:09,933 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:09,933 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:09,946 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:09,946 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:09,946 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:09,946 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. started... 2023-04-12 15:54:09,947 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. as already flushing 2023-04-12 15:54:09,947 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:09,947 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:09,947 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:09,947 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,948 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:09,948 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:09,949 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,949 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:09,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:09,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:09,950 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:09,950 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:09,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:09,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:09,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:09,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:09,952 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:09,952 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16230531[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:09,952 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:09,952 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:09,953 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:09,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:09,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:09,954 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:09,954 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:09,954 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:09,954 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,954 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:09,958 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,958 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:09,958 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:09,958 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:09,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:09,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:09,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:09,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:09,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:09,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:09,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:09,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:09,962 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:09,962 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:09,962 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:09,962 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:09,962 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:09,962 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:09,966 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:09,966 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:09,966 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:09,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:09,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:09,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:09,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:09,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:09,967 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:09,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:09,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:09,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:09,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:09,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:09,968 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:09,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:09,973 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:09,973 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:09,973 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:09,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:09,973 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:09,974 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:09,974 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:09,974 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:09,974 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:09,974 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:09,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:09,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:09,974 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:09,974 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:09,974 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:09,974 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:09,974 DEBUG [Thread-831] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:09,975 DEBUG [Thread-831] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:10,075 DEBUG [Thread-831] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:10,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:10,078 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:10,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:10,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:10,081 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:10,081 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:10,081 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:10,081 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:10,082 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:10,082 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:10,088 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:10,088 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:10,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:10,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:10,089 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:10,089 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:10,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:10,089 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:10,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:10,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:10,090 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:10,092 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:10,092 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:10,092 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:10,092 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:10,092 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:10,092 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:10,092 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:10,092 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:10,092 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. started... 2023-04-12 15:54:10,093 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. as already flushing 2023-04-12 15:54:10,093 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:10,093 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:10,093 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:10,093 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:10,093 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:10,093 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:10,094 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:10,094 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:10,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:10,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:10,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:10,095 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:10,095 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:10,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:10,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:10,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:10,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:10,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:10,096 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:10,096 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:10,096 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d2a39d[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:10,096 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:10,097 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:10,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:10,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:10,098 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:10,098 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:10,098 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:10,098 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:10,098 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:10,100 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:10,100 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:10,100 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:10,100 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:10,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:10,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:10,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:10,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:10,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:10,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:10,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:10,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:10,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:10,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:10,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:10,103 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:10,103 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:10,103 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:10,103 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:10,103 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:10,103 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:10,109 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:10,109 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:10,109 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:10,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:10,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:10,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:10,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:10,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:10,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:10,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:10,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:10,110 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:10,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:10,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:10,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:10,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:10,119 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:10,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:10,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:10,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:10,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:10,123 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:10,123 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:10,123 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:10,123 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:10,123 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:10,123 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:10,123 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:10,123 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:10,123 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:10,123 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:10,123 DEBUG [Thread-831] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:10,123 DEBUG [Thread-831] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:10,124 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:10,124 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:10,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:10,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:10,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:10,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:10,127 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:10,223 DEBUG [Thread-831] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:10,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:10,226 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:10,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:10,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:10,227 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:10,227 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:10,227 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:10,227 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:10,228 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:10,228 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:10,229 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:10,229 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:10,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:10,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:10,230 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:10,230 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:10,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:10,230 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:10,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:10,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:10,230 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:10,231 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:10,231 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:10,231 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:10,231 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:10,231 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:10,232 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:10,232 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:10,232 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:10,232 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. started... 2023-04-12 15:54:10,232 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. as already flushing 2023-04-12 15:54:10,232 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:10,232 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:10,232 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:10,232 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:10,232 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:10,232 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:10,234 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:10,234 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:10,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:10,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:10,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:10,234 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:10,234 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:10,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:10,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:10,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:10,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:10,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:10,236 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:10,236 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e4002c1[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:10,236 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:10,236 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:10,238 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:10,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:10,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:10,238 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:10,238 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:10,238 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:10,239 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:10,238 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:10,240 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:10,240 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:10,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:10,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:10,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:10,240 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:10,240 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:10,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:10,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:10,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:10,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:10,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:10,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:10,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:10,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:10,242 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:10,242 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:10,243 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:10,243 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:10,243 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:10,243 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:10,244 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:10,244 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:10,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:10,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:10,244 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:10,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:10,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:10,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:10,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:10,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:10,245 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:10,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:10,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:10,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:10,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:10,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:10,246 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:10,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:10,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:10,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:10,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:10,249 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:10,249 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:10,249 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:10,249 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:10,249 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:10,249 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:10,249 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:10,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:10,249 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:10,249 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:10,249 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:10,249 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:10,249 DEBUG [Thread-831] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:10,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:10,249 DEBUG [Thread-831] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:10,249 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:10,249 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:10,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:10,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:10,263 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/C/85933e793f944cde9c58b7cfdd679f30 2023-04-12 15:54:10,268 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/A/241794289536461ab44241ae2b5cd066 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/241794289536461ab44241ae2b5cd066 2023-04-12 15:54:10,273 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/241794289536461ab44241ae2b5cd066, entries=150, sequenceid=14, filesize=25.1 K 2023-04-12 15:54:10,273 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/B/890822e2f7634ea9b23c646406f620ce as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/890822e2f7634ea9b23c646406f620ce 2023-04-12 15:54:10,279 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/890822e2f7634ea9b23c646406f620ce, entries=150, sequenceid=14, filesize=11.6 K 2023-04-12 15:54:10,280 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/C/85933e793f944cde9c58b7cfdd679f30 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/85933e793f944cde9c58b7cfdd679f30 2023-04-12 15:54:10,286 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/85933e793f944cde9c58b7cfdd679f30, entries=150, sequenceid=14, filesize=11.6 K 2023-04-12 15:54:10,289 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for ca92eda5f752c4c8f5e2f01f66bc73d8 in 1061ms, sequenceid=14, compaction requested=false 2023-04-12 15:54:10,289 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ca92eda5f752c4c8f5e2f01f66bc73d8: 2023-04-12 15:54:10,349 DEBUG [Thread-831] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:10,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:10,352 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:10,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:10,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:10,353 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:10,353 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:10,353 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:10,354 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:10,354 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:10,354 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:10,357 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:10,357 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:10,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:10,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:10,357 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:10,358 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:10,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:10,358 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:10,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:10,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:10,358 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:10,359 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:10,359 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:10,359 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:10,359 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:10,359 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:10,359 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:10,359 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:10,359 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:10,359 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. started... 2023-04-12 15:54:10,359 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ca92eda5f752c4c8f5e2f01f66bc73d8 3/3 column families, dataSize=134.18 KB heapSize=375.61 KB 2023-04-12 15:54:10,360 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca92eda5f752c4c8f5e2f01f66bc73d8, store=A 2023-04-12 15:54:10,360 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:54:10,360 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca92eda5f752c4c8f5e2f01f66bc73d8, store=B 2023-04-12 15:54:10,360 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:54:10,360 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca92eda5f752c4c8f5e2f01f66bc73d8, store=C 2023-04-12 15:54:10,360 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:54:10,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on ca92eda5f752c4c8f5e2f01f66bc73d8 2023-04-12 15:54:10,411 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. as already flushing 2023-04-12 15:54:10,412 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230412d5e95cc3ecf940cfb2ea1c1c754691a2 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412d5e95cc3ecf940cfb2ea1c1c754691a2 2023-04-12 15:54:10,424 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:10,424 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:10,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34094 deadline: 1681314910415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:10,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34066 deadline: 1681314910416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:10,425 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:10,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34118 deadline: 1681314910422, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:10,425 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:10,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34122 deadline: 1681314910425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:10,425 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:10,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34134 deadline: 1681314910425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:10,439 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=49, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/A/00a4c9a026ca43578c8fe7ed865f489e 2023-04-12 15:54:10,527 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:10,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34066 deadline: 1681314910526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:10,527 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:10,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34094 deadline: 1681314910526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:10,528 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:10,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34122 deadline: 1681314910527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:10,528 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:10,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34134 deadline: 1681314910527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:10,730 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:10,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34094 deadline: 1681314910729, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:10,731 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:10,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34066 deadline: 1681314910729, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:10,732 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:10,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34122 deadline: 1681314910730, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:10,732 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:10,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34134 deadline: 1681314910730, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:10,866 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=49 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/B/22cdd7ba8a3c4e3b8a139e86d83e2f2b 2023-04-12 15:54:10,922 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=49 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/C/da84f2ab521b4d89bedc4299dd71deba 2023-04-12 15:54:10,930 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/A/00a4c9a026ca43578c8fe7ed865f489e as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/00a4c9a026ca43578c8fe7ed865f489e 2023-04-12 15:54:10,935 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/00a4c9a026ca43578c8fe7ed865f489e, entries=150, sequenceid=49, filesize=25.1 K 2023-04-12 15:54:10,936 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/B/22cdd7ba8a3c4e3b8a139e86d83e2f2b as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/22cdd7ba8a3c4e3b8a139e86d83e2f2b 2023-04-12 15:54:10,942 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/22cdd7ba8a3c4e3b8a139e86d83e2f2b, entries=150, sequenceid=49, filesize=11.6 K 2023-04-12 15:54:10,943 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/C/da84f2ab521b4d89bedc4299dd71deba as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/da84f2ab521b4d89bedc4299dd71deba 2023-04-12 15:54:10,948 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/da84f2ab521b4d89bedc4299dd71deba, entries=150, sequenceid=49, filesize=11.6 K 2023-04-12 15:54:10,949 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~375.56 KB/384576, currentSize=53.67 KB/54960 for ca92eda5f752c4c8f5e2f01f66bc73d8 in 590ms, sequenceid=49, compaction requested=false 2023-04-12 15:54:10,950 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ca92eda5f752c4c8f5e2f01f66bc73d8: 2023-04-12 15:54:10,950 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:10,950 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:10,950 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:10,950 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:10,950 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:10,950 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:10,952 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:10,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:10,952 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:10,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:10,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:10,952 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:10,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:10,952 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:10,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:10,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:10,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:10,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:10,956 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:10,956 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@496f2d00[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:10,956 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:10,956 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:10,960 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:10,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:10,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:10,960 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:10,960 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:10,960 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:10,961 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:10,961 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:10,963 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:10,963 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:10,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:10,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:10,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:10,963 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:10,963 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:10,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:10,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:10,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:10,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:10,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:10,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:10,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:10,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:10,968 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:10,968 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:10,968 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:10,968 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:10,968 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:10,968 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:10,970 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:10,970 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:10,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:10,970 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:10,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:10,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:10,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:10,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:10,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:10,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:10,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:10,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:10,972 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:10,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:10,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:10,975 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:10,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:10,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:10,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:10,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:10,980 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:10,980 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:10,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:10,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:10,981 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:10,981 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:10,981 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:10,981 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:10,981 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:10,981 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:10,981 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:10,981 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:10,981 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:10,981 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:10,981 DEBUG [Thread-831] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:10,981 DEBUG [Thread-831] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:10,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:10,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:10,982 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:11,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on ca92eda5f752c4c8f5e2f01f66bc73d8 2023-04-12 15:54:11,034 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ca92eda5f752c4c8f5e2f01f66bc73d8 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-12 15:54:11,034 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca92eda5f752c4c8f5e2f01f66bc73d8, store=A 2023-04-12 15:54:11,035 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:11,035 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca92eda5f752c4c8f5e2f01f66bc73d8, store=B 2023-04-12 15:54:11,035 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:11,035 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca92eda5f752c4c8f5e2f01f66bc73d8, store=C 2023-04-12 15:54:11,035 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:11,055 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:11,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34134 deadline: 1681314911049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:11,057 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:11,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34066 deadline: 1681314911054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:11,057 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:11,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34094 deadline: 1681314911055, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:11,058 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:11,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34122 deadline: 1681314911056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:11,065 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230412e7c60fa217ef46b1927588fcf3c28de2 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412e7c60fa217ef46b1927588fcf3c28de2 2023-04-12 15:54:11,081 DEBUG [Thread-831] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:11,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:11,084 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:11,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:11,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:11,092 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:11,092 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:11,092 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:11,092 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:11,093 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:11,093 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:11,095 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:11,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:11,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:11,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:11,096 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:11,096 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:11,096 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:11,096 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:11,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:11,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:11,097 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:11,097 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:11,097 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:11,097 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:11,097 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:11,097 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:11,100 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:11,100 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:11,100 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. started... 2023-04-12 15:54:11,100 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:11,100 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. as already flushing 2023-04-12 15:54:11,100 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:11,100 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:11,100 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:11,100 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:11,100 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:11,100 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:11,102 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:11,102 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:11,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:11,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:11,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:11,102 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:11,102 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:11,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:11,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:11,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:11,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:11,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:11,105 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:11,105 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47ebb391[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:11,105 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:11,105 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:11,106 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:11,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:11,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:11,106 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:11,107 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:11,107 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:11,107 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:11,107 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:11,108 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:11,108 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:11,108 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:11,108 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:11,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:11,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:11,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:11,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:11,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:11,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:11,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:11,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:11,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:11,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:11,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:11,112 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:11,112 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:11,112 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:11,112 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:11,112 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:11,112 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:11,115 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:11,115 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:11,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:11,115 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:11,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:11,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:11,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:11,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:11,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:11,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:11,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:11,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:11,116 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:11,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:11,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:11,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:11,118 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:11,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:11,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:11,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:11,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:11,120 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:11,120 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:11,120 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:11,120 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:11,120 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:11,120 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:11,120 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:11,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:11,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:11,120 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:11,120 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:11,120 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:11,121 DEBUG [Thread-831] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:11,120 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:11,120 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:11,121 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:11,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:11,121 DEBUG [Thread-831] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:11,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:11,158 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:11,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34134 deadline: 1681314911157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:11,159 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:11,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34066 deadline: 1681314911158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:11,159 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:11,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34094 deadline: 1681314911158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:11,160 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:11,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34122 deadline: 1681314911159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:11,221 DEBUG [Thread-831] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:11,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:11,223 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:11,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:11,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:11,224 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:11,225 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:11,225 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:11,225 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:11,225 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:11,225 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:11,226 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:11,226 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:11,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:11,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:11,227 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:11,227 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:11,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:11,227 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:11,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:11,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:11,228 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:11,228 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:11,228 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:11,228 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:11,228 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:11,228 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:11,228 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:11,228 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:11,228 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:11,229 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. started... 2023-04-12 15:54:11,229 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. as already flushing 2023-04-12 15:54:11,229 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:11,229 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:11,229 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:11,229 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:11,229 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:11,229 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:11,230 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:11,230 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:11,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:11,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:11,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:11,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:11,231 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:11,231 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:11,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:11,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:11,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:11,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:11,232 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:11,232 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:11,232 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53b13eba[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:11,232 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:11,237 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:11,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:11,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:11,237 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:11,237 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:11,237 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:11,237 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:11,237 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:11,239 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:11,239 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:11,239 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:11,239 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:11,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:11,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:11,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:11,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:11,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:11,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:11,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:11,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:11,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:11,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:11,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:11,241 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:11,241 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:11,242 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:11,242 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:11,242 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:11,242 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:11,243 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:11,243 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:11,243 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:11,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:11,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:11,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:11,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:11,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:11,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:11,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:11,244 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:11,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:11,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:11,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:11,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:11,245 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:11,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:11,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:11,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:11,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:11,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:11,247 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:11,247 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:11,247 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:11,247 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:11,248 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:11,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:11,248 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:11,248 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:11,248 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:11,248 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:11,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:11,248 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:11,248 DEBUG [Thread-831] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:11,248 DEBUG [Thread-831] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:11,248 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:11,248 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:11,248 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:11,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:11,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:11,348 DEBUG [Thread-831] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:11,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:11,352 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:11,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:11,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:11,354 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:11,354 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:11,354 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:11,354 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:11,354 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:11,354 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:11,356 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:11,356 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:11,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:11,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:11,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:11,356 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:11,356 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:11,357 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:11,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:11,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:11,357 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:11,357 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:11,357 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:11,357 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:11,357 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:11,357 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:11,358 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:11,358 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:11,358 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:11,358 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. started... 2023-04-12 15:54:11,358 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. as already flushing 2023-04-12 15:54:11,358 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:11,358 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:11,358 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:11,358 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:11,358 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:11,358 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:11,361 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:11,361 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:11,361 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:11,363 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:11,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34134 deadline: 1681314911360, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:11,364 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:11,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34066 deadline: 1681314911360, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:11,364 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:11,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:11,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:11,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:11,364 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:11,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34094 deadline: 1681314911360, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:11,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:11,365 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:11,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:11,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34122 deadline: 1681314911363, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:11,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:11,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:11,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:11,366 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:11,366 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d697807[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:11,366 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:11,366 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:11,368 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:11,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:11,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:11,369 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:11,369 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:11,369 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:11,369 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:11,369 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:11,370 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:11,370 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:11,370 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:11,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:11,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:11,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:11,370 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:11,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:11,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:11,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:11,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:11,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:11,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:11,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:11,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:11,373 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:11,373 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:11,373 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:11,373 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:11,373 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:11,373 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:11,374 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:11,374 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:11,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:11,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:11,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:11,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:11,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:11,374 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:11,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:11,375 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:11,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:11,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:11,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:11,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:11,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:11,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:11,377 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:11,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:11,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:11,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:11,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:11,383 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:11,383 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:11,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:11,383 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:11,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:11,383 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:11,383 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:11,383 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:11,383 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:11,383 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:11,384 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:11,384 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:11,384 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:11,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:11,384 DEBUG [Thread-831] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:11,384 DEBUG [Thread-831] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:11,384 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:11,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:11,384 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:11,434 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:11,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34118 deadline: 1681314911431, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:11,484 DEBUG [Thread-831] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:11,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:11,487 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=61, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/A/5f9d968d7a5c4ce5afc405b82e01abe9 2023-04-12 15:54:11,488 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:11,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:11,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:11,491 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:11,491 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:11,491 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:11,491 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:11,491 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:11,491 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:11,493 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:11,493 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:11,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:11,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:11,493 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:11,494 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:11,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:11,494 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:11,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:11,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:11,495 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:11,495 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:11,495 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:11,496 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:11,496 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:11,496 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:11,496 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:11,496 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:11,496 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:11,496 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. started... 2023-04-12 15:54:11,496 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. as already flushing 2023-04-12 15:54:11,497 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:11,497 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:11,497 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:11,497 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:11,497 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:11,497 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:11,500 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:11,500 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:11,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:11,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:11,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:11,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:11,501 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:11,501 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:11,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:11,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:11,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:11,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:11,502 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:11,502 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6dde5987[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:11,502 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:11,502 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:11,504 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:11,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:11,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:11,504 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:11,504 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:11,504 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:11,504 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:11,504 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:11,506 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=61 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/B/46851ff41db6423aaa9e6579f679b062 2023-04-12 15:54:11,507 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:11,507 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:11,507 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:11,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:11,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:11,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:11,507 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:11,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:11,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:11,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:11,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:11,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:11,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:11,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:11,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:11,510 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:11,511 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:11,511 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:11,511 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:11,511 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:11,511 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:11,513 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:11,513 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:11,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:11,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:11,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:11,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:11,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:11,513 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:11,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:11,514 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:11,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:11,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:11,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:11,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:11,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:11,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:11,517 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:11,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:11,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:11,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:11,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:11,520 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:11,520 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:11,520 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:11,520 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:11,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:11,520 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:11,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:11,520 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:11,520 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:11,521 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:11,521 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:11,521 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:11,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:11,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:11,521 DEBUG [Thread-831] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:11,521 DEBUG [Thread-831] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:11,521 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:11,521 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:11,527 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=61 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/C/110ced44107f4298a942f0f97c4a9666 2023-04-12 15:54:11,532 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/A/5f9d968d7a5c4ce5afc405b82e01abe9 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/5f9d968d7a5c4ce5afc405b82e01abe9 2023-04-12 15:54:11,538 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/5f9d968d7a5c4ce5afc405b82e01abe9, entries=150, sequenceid=61, filesize=25.1 K 2023-04-12 15:54:11,539 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/B/46851ff41db6423aaa9e6579f679b062 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/46851ff41db6423aaa9e6579f679b062 2023-04-12 15:54:11,545 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/46851ff41db6423aaa9e6579f679b062, entries=150, sequenceid=61, filesize=11.6 K 2023-04-12 15:54:11,546 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/C/110ced44107f4298a942f0f97c4a9666 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/110ced44107f4298a942f0f97c4a9666 2023-04-12 15:54:11,553 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/110ced44107f4298a942f0f97c4a9666, entries=150, sequenceid=61, filesize=11.6 K 2023-04-12 15:54:11,564 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for ca92eda5f752c4c8f5e2f01f66bc73d8 in 530ms, sequenceid=61, compaction requested=true 2023-04-12 15:54:11,564 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ca92eda5f752c4c8f5e2f01f66bc73d8: 2023-04-12 15:54:11,564 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 15:54:11,564 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:54:11,565 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 15:54:11,565 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:54:11,566 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:54:11,566 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 15:54:11,566 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1912): ca92eda5f752c4c8f5e2f01f66bc73d8/B is initiating minor compaction (all files) 2023-04-12 15:54:11,566 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 77238 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 15:54:11,567 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1912): ca92eda5f752c4c8f5e2f01f66bc73d8/A is initiating minor compaction (all files) 2023-04-12 15:54:11,567 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ca92eda5f752c4c8f5e2f01f66bc73d8/A in TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:11,567 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/241794289536461ab44241ae2b5cd066, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/00a4c9a026ca43578c8fe7ed865f489e, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/5f9d968d7a5c4ce5afc405b82e01abe9] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp, totalSize=75.4 K 2023-04-12 15:54:11,566 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ca92eda5f752c4c8f5e2f01f66bc73d8/B in TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:11,567 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/890822e2f7634ea9b23c646406f620ce, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/22cdd7ba8a3c4e3b8a139e86d83e2f2b, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/46851ff41db6423aaa9e6579f679b062] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp, totalSize=34.8 K 2023-04-12 15:54:11,568 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 241794289536461ab44241ae2b5cd066, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1681314849218 2023-04-12 15:54:11,568 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 00a4c9a026ca43578c8fe7ed865f489e, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1681314849278 2023-04-12 15:54:11,570 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 5f9d968d7a5c4ce5afc405b82e01abe9, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1681314850422 2023-04-12 15:54:11,570 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 890822e2f7634ea9b23c646406f620ce, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1681314849218 2023-04-12 15:54:11,570 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 22cdd7ba8a3c4e3b8a139e86d83e2f2b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1681314849278 2023-04-12 15:54:11,571 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 46851ff41db6423aaa9e6579f679b062, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1681314850422 2023-04-12 15:54:11,583 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] throttle.PressureAwareThroughputController(145): ca92eda5f752c4c8f5e2f01f66bc73d8#B#compaction#111 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:54:11,600 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] throttle.PressureAwareThroughputController(145): ca92eda5f752c4c8f5e2f01f66bc73d8#A#compaction#112 average throughput is 1.23 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:54:11,621 DEBUG [Thread-831] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:11,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:11,623 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/B/b6eb09dc371b4a5fa98c56978e88aaa0 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/b6eb09dc371b4a5fa98c56978e88aaa0 2023-04-12 15:54:11,624 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:11,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:11,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:11,628 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:11,628 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:11,628 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:11,628 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:11,629 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:11,629 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ca92eda5f752c4c8f5e2f01f66bc73d8/B of ca92eda5f752c4c8f5e2f01f66bc73d8 into b6eb09dc371b4a5fa98c56978e88aaa0(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:54:11,629 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:11,629 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ca92eda5f752c4c8f5e2f01f66bc73d8: 2023-04-12 15:54:11,629 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8., storeName=ca92eda5f752c4c8f5e2f01f66bc73d8/B, priority=13, startTime=1681314851564; duration=0sec 2023-04-12 15:54:11,629 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:54:11,629 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 15:54:11,630 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:11,630 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:11,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:11,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:11,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:11,631 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:11,631 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:11,631 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 15:54:11,632 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1912): ca92eda5f752c4c8f5e2f01f66bc73d8/C is initiating minor compaction (all files) 2023-04-12 15:54:11,632 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:11,632 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ca92eda5f752c4c8f5e2f01f66bc73d8/C in TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:11,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:11,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:11,632 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:11,633 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/85933e793f944cde9c58b7cfdd679f30, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/da84f2ab521b4d89bedc4299dd71deba, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/110ced44107f4298a942f0f97c4a9666] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp, totalSize=34.8 K 2023-04-12 15:54:11,633 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:11,633 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:11,633 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:11,633 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:11,633 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:11,633 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 85933e793f944cde9c58b7cfdd679f30, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1681314849218 2023-04-12 15:54:11,633 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting da84f2ab521b4d89bedc4299dd71deba, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1681314849278 2023-04-12 15:54:11,634 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 110ced44107f4298a942f0f97c4a9666, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1681314850422 2023-04-12 15:54:11,634 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:11,634 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:11,634 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:11,634 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. started... 2023-04-12 15:54:11,634 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ca92eda5f752c4c8f5e2f01f66bc73d8 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-12 15:54:11,635 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca92eda5f752c4c8f5e2f01f66bc73d8, store=A 2023-04-12 15:54:11,635 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:11,635 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca92eda5f752c4c8f5e2f01f66bc73d8, store=B 2023-04-12 15:54:11,635 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:11,635 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca92eda5f752c4c8f5e2f01f66bc73d8, store=C 2023-04-12 15:54:11,635 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:11,643 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] throttle.PressureAwareThroughputController(145): ca92eda5f752c4c8f5e2f01f66bc73d8#C#compaction#114 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:54:11,654 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/C/e0f9f5ef76834f83a88df39bfdf88372 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/e0f9f5ef76834f83a88df39bfdf88372 2023-04-12 15:54:11,660 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ca92eda5f752c4c8f5e2f01f66bc73d8/C of ca92eda5f752c4c8f5e2f01f66bc73d8 into e0f9f5ef76834f83a88df39bfdf88372(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:54:11,661 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ca92eda5f752c4c8f5e2f01f66bc73d8: 2023-04-12 15:54:11,661 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8., storeName=ca92eda5f752c4c8f5e2f01f66bc73d8/C, priority=13, startTime=1681314851566; duration=0sec 2023-04-12 15:54:11,661 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:54:11,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on ca92eda5f752c4c8f5e2f01f66bc73d8 2023-04-12 15:54:11,668 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. as already flushing 2023-04-12 15:54:11,675 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:11,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34134 deadline: 1681314911673, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:11,678 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:11,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34122 deadline: 1681314911674, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:11,679 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:11,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34094 deadline: 1681314911675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:11,679 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:11,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34066 deadline: 1681314911675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:11,777 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:11,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34134 deadline: 1681314911777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:11,781 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:11,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34122 deadline: 1681314911779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:11,782 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:11,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34094 deadline: 1681314911780, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:11,782 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:11,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34066 deadline: 1681314911781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:11,980 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:11,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34134 deadline: 1681314911980, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:11,984 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:11,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34094 deadline: 1681314911983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:11,984 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:11,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34066 deadline: 1681314911983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:11,985 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:11,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34122 deadline: 1681314911983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:12,062 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304120926341ac65042f7acd1f7577cdba08c to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304120926341ac65042f7acd1f7577cdba08c 2023-04-12 15:54:12,077 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/A/8d3fbd974ffd4340b479f0bcf22e0b01 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/8d3fbd974ffd4340b479f0bcf22e0b01 2023-04-12 15:54:12,088 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ca92eda5f752c4c8f5e2f01f66bc73d8/A of ca92eda5f752c4c8f5e2f01f66bc73d8 into 8d3fbd974ffd4340b479f0bcf22e0b01(size=25.2 K), total size for store is 25.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:54:12,088 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ca92eda5f752c4c8f5e2f01f66bc73d8: 2023-04-12 15:54:12,088 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8., storeName=ca92eda5f752c4c8f5e2f01f66bc73d8/A, priority=13, startTime=1681314851564; duration=0sec 2023-04-12 15:54:12,088 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:54:12,105 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=82, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/A/5abb434e890b43f8b48a8dfc06182063 2023-04-12 15:54:12,284 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:12,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34134 deadline: 1681314912283, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:12,286 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:12,286 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:12,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34094 deadline: 1681314912285, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:12,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34066 deadline: 1681314912285, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:12,289 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:12,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34122 deadline: 1681314912287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:12,544 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/B/9f1c46d5ce854a2b88438c74d405d1e7 2023-04-12 15:54:12,562 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/C/4ed8ee44d96b4498869b3612f17a7985 2023-04-12 15:54:12,571 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/A/5abb434e890b43f8b48a8dfc06182063 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/5abb434e890b43f8b48a8dfc06182063 2023-04-12 15:54:12,576 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/5abb434e890b43f8b48a8dfc06182063, entries=150, sequenceid=82, filesize=25.1 K 2023-04-12 15:54:12,578 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/B/9f1c46d5ce854a2b88438c74d405d1e7 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/9f1c46d5ce854a2b88438c74d405d1e7 2023-04-12 15:54:12,582 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/9f1c46d5ce854a2b88438c74d405d1e7, entries=150, sequenceid=82, filesize=11.6 K 2023-04-12 15:54:12,583 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/C/4ed8ee44d96b4498869b3612f17a7985 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/4ed8ee44d96b4498869b3612f17a7985 2023-04-12 15:54:12,589 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/4ed8ee44d96b4498869b3612f17a7985, entries=150, sequenceid=82, filesize=11.6 K 2023-04-12 15:54:12,589 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for ca92eda5f752c4c8f5e2f01f66bc73d8 in 955ms, sequenceid=82, compaction requested=false 2023-04-12 15:54:12,590 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ca92eda5f752c4c8f5e2f01f66bc73d8: 2023-04-12 15:54:12,590 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:12,590 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:12,590 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:12,590 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:12,590 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:12,590 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:12,590 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-12 15:54:12,593 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:12,593 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:12,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:12,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:12,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:12,593 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:12,593 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:12,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:12,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:12,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:12,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:12,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:12,596 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:12,597 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77b30896[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:12,597 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:12,597 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:12,598 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:12,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:12,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:12,598 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:12,598 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:12,599 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:12,599 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:12,599 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:12,600 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:12,600 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:12,600 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:12,600 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:12,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:12,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:12,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:12,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:12,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:12,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:12,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:12,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:12,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:12,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:12,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:12,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:12,606 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:12,606 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:12,606 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:12,606 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:12,606 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:12,609 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:12,609 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:12,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:12,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:12,609 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:12,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:12,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:12,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:12,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:12,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:12,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:12,609 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:12,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:12,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:12,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:12,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:12,611 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:12,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:12,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:12,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:12,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:12,614 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:12,614 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:12,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:12,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:12,614 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:12,614 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:12,614 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:12,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:12,615 DEBUG [Thread-831] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:12,615 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:12,615 DEBUG [Thread-831] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:12,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:12,615 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:12,615 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:12,615 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:12,615 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:12,615 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:12,615 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:12,615 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:12,715 DEBUG [Thread-831] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:12,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:12,718 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:12,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:12,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:12,719 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:12,720 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:12,720 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:12,720 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:12,720 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:12,720 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:12,721 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:12,721 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:12,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:12,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:12,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:12,722 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:12,722 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:12,722 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:12,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:12,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:12,722 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:12,723 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:12,723 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:12,723 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:12,723 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:12,723 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:12,724 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:12,724 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:12,724 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:12,725 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. started... 2023-04-12 15:54:12,725 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ca92eda5f752c4c8f5e2f01f66bc73d8 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-12 15:54:12,725 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca92eda5f752c4c8f5e2f01f66bc73d8, store=A 2023-04-12 15:54:12,725 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:12,725 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca92eda5f752c4c8f5e2f01f66bc73d8, store=B 2023-04-12 15:54:12,726 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:12,726 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca92eda5f752c4c8f5e2f01f66bc73d8, store=C 2023-04-12 15:54:12,726 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:12,753 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230412a72d8d2cb4814a39a29747baf3ae860e to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412a72d8d2cb4814a39a29747baf3ae860e 2023-04-12 15:54:12,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on ca92eda5f752c4c8f5e2f01f66bc73d8 2023-04-12 15:54:12,792 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. as already flushing 2023-04-12 15:54:12,812 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:12,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34094 deadline: 1681314912810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:12,814 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:12,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34134 deadline: 1681314912811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:12,814 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:12,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34066 deadline: 1681314912811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:12,814 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:12,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34122 deadline: 1681314912812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:12,916 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:12,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34094 deadline: 1681314912913, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:12,919 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:12,919 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:12,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34134 deadline: 1681314912915, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:12,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34122 deadline: 1681314912915, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:12,919 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:12,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34066 deadline: 1681314912915, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:13,118 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:13,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34094 deadline: 1681314913118, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:13,124 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:13,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34066 deadline: 1681314913122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:13,124 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:13,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34122 deadline: 1681314913122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:13,124 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:13,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34134 deadline: 1681314913122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:13,165 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=96, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/A/da539482f9214cc69b85aaee0c00fecc 2023-04-12 15:54:13,178 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=96 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/B/841a0cf8efa64001929f1be245bbdbb0 2023-04-12 15:54:13,421 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:13,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34094 deadline: 1681314913420, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:13,427 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:13,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34066 deadline: 1681314913425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:13,427 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:13,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34122 deadline: 1681314913425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:13,428 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:13,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34134 deadline: 1681314913426, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:13,447 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:13,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34118 deadline: 1681314913445, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:13,450 DEBUG [Thread-829] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4168 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8., hostname=jenkins-hbase4.apache.org,38905,1681314818739, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-12 15:54:13,590 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=96 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/C/93baa4de2d5042c2a2bd03d3ea36df9c 2023-04-12 15:54:13,600 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/A/da539482f9214cc69b85aaee0c00fecc as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/da539482f9214cc69b85aaee0c00fecc 2023-04-12 15:54:13,613 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/da539482f9214cc69b85aaee0c00fecc, entries=150, sequenceid=96, filesize=25.1 K 2023-04-12 15:54:13,615 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/B/841a0cf8efa64001929f1be245bbdbb0 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/841a0cf8efa64001929f1be245bbdbb0 2023-04-12 15:54:13,624 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/841a0cf8efa64001929f1be245bbdbb0, entries=150, sequenceid=96, filesize=11.6 K 2023-04-12 15:54:13,625 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/C/93baa4de2d5042c2a2bd03d3ea36df9c as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/93baa4de2d5042c2a2bd03d3ea36df9c 2023-04-12 15:54:13,631 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/93baa4de2d5042c2a2bd03d3ea36df9c, entries=150, sequenceid=96, filesize=11.6 K 2023-04-12 15:54:13,633 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for ca92eda5f752c4c8f5e2f01f66bc73d8 in 908ms, sequenceid=96, compaction requested=true 2023-04-12 15:54:13,633 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ca92eda5f752c4c8f5e2f01f66bc73d8: 2023-04-12 15:54:13,633 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:13,633 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:13,633 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:13,633 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:13,633 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:13,633 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:13,635 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:13,635 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:13,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:13,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:13,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:13,635 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:13,635 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:13,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:13,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:13,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:13,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:13,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:13,638 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:13,638 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2dadbc34[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:13,638 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:13,638 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:13,639 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:13,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:13,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:13,639 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:13,639 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:13,639 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:13,639 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:13,639 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:13,641 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:13,641 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:13,641 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:13,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:13,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:13,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:13,641 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:13,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:13,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:13,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:13,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:13,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:13,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:13,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:13,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:13,644 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:13,644 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:13,644 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:13,644 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:13,644 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:13,644 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:13,647 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:13,647 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:13,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:13,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:13,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:13,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:13,647 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:13,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:13,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:13,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:13,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:13,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:13,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:13,649 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:13,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:13,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:13,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:13,650 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:13,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:13,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:13,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:13,655 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:13,655 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:13,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:13,655 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:13,655 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:13,655 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:13,655 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:13,655 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:13,655 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:13,655 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:13,655 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:13,655 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:13,655 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:13,655 DEBUG [Thread-831] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:13,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:13,655 DEBUG [Thread-831] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:13,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:13,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:13,756 DEBUG [Thread-831] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:13,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:13,759 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:13,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:13,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:13,765 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:13,765 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:13,765 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:13,765 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:13,766 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:13,766 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:13,767 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:13,767 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:13,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:13,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:13,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:13,768 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:13,768 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:13,768 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:13,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:13,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:13,768 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:13,769 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:13,769 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:13,769 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:13,769 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:13,769 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:13,773 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:13,773 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:13,773 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:13,773 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. started... 2023-04-12 15:54:13,773 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ca92eda5f752c4c8f5e2f01f66bc73d8 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-12 15:54:13,773 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca92eda5f752c4c8f5e2f01f66bc73d8, store=A 2023-04-12 15:54:13,773 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:13,773 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca92eda5f752c4c8f5e2f01f66bc73d8, store=B 2023-04-12 15:54:13,773 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:13,774 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca92eda5f752c4c8f5e2f01f66bc73d8, store=C 2023-04-12 15:54:13,781 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:13,843 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230412f8614ba0066c4a13a279be61be8c6dfc to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412f8614ba0066c4a13a279be61be8c6dfc 2023-04-12 15:54:13,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on ca92eda5f752c4c8f5e2f01f66bc73d8 2023-04-12 15:54:13,930 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. as already flushing 2023-04-12 15:54:13,942 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:13,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34134 deadline: 1681314913938, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:13,943 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:13,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34122 deadline: 1681314913940, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:13,943 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:13,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34094 deadline: 1681314913940, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:13,943 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:13,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34066 deadline: 1681314913942, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:14,045 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:14,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34094 deadline: 1681314914045, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:14,045 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:14,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34134 deadline: 1681314914045, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:14,045 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:14,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34122 deadline: 1681314914045, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:14,047 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:14,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34066 deadline: 1681314914045, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:14,250 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:14,250 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:14,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34094 deadline: 1681314914246, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:14,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34134 deadline: 1681314914247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:14,251 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:14,251 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:14,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34122 deadline: 1681314914248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:14,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34066 deadline: 1681314914248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:14,258 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=116, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/A/62b43f9b9d574e0882527b21f5e022cc 2023-04-12 15:54:14,276 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=116 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/B/4bae1d6cd92f4824844d04d8edbd5aa6 2023-04-12 15:54:14,290 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=116 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/C/5611fc74272649b39d04a33911779ca3 2023-04-12 15:54:14,298 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/A/62b43f9b9d574e0882527b21f5e022cc as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/62b43f9b9d574e0882527b21f5e022cc 2023-04-12 15:54:14,304 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/62b43f9b9d574e0882527b21f5e022cc, entries=150, sequenceid=116, filesize=25.1 K 2023-04-12 15:54:14,307 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/B/4bae1d6cd92f4824844d04d8edbd5aa6 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/4bae1d6cd92f4824844d04d8edbd5aa6 2023-04-12 15:54:14,313 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/4bae1d6cd92f4824844d04d8edbd5aa6, entries=150, sequenceid=116, filesize=11.6 K 2023-04-12 15:54:14,314 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/C/5611fc74272649b39d04a33911779ca3 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/5611fc74272649b39d04a33911779ca3 2023-04-12 15:54:14,320 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/5611fc74272649b39d04a33911779ca3, entries=150, sequenceid=116, filesize=11.6 K 2023-04-12 15:54:14,321 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for ca92eda5f752c4c8f5e2f01f66bc73d8 in 548ms, sequenceid=116, compaction requested=true 2023-04-12 15:54:14,321 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ca92eda5f752c4c8f5e2f01f66bc73d8: 2023-04-12 15:54:14,321 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:14,325 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:14,325 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:14,326 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:14,326 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:14,326 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:14,334 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:14,334 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:14,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:14,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:14,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:14,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:14,334 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:14,334 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:14,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:14,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:14,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:14,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:14,335 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:14,336 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b1240c6[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:14,336 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:14,336 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:14,337 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:14,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:14,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:14,337 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:14,337 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:14,337 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:14,337 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:14,337 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:14,338 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:14,338 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:14,338 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:14,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:14,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:14,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:14,338 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:14,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:14,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:14,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:14,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:14,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:14,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:14,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:14,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:14,341 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:14,341 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:14,341 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:14,341 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:14,341 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:14,341 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:14,342 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:14,342 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:14,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:14,342 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:14,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:14,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:14,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:14,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:14,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:14,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:14,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:14,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:14,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:14,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:14,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:14,344 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:14,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:14,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:14,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:14,345 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:14,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:14,352 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:14,352 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:14,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:14,352 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:14,352 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:14,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:14,352 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:14,352 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:14,353 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:14,353 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:14,353 DEBUG [Thread-831] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:14,353 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:14,353 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:14,353 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:14,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:14,353 DEBUG [Thread-831] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:14,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:14,353 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:14,453 DEBUG [Thread-831] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:14,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:14,458 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:14,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:14,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:14,463 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:14,463 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:14,463 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:14,463 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:14,464 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:14,464 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:14,465 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:14,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:14,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:14,465 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:14,466 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:14,466 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:14,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:14,466 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:14,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:14,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:14,467 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:14,467 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:14,467 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:14,467 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:14,467 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:14,467 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:14,476 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:14,476 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:14,476 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:14,476 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. started... 2023-04-12 15:54:14,476 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ca92eda5f752c4c8f5e2f01f66bc73d8 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-12 15:54:14,476 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca92eda5f752c4c8f5e2f01f66bc73d8, store=A 2023-04-12 15:54:14,476 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:14,477 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca92eda5f752c4c8f5e2f01f66bc73d8, store=B 2023-04-12 15:54:14,478 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:14,478 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca92eda5f752c4c8f5e2f01f66bc73d8, store=C 2023-04-12 15:54:14,478 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:14,514 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304129a3aeaf36ced4b66af71b5813c9fd285 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304129a3aeaf36ced4b66af71b5813c9fd285 2023-04-12 15:54:14,522 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=128, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/A/374549cd7be54c8ca75b1c947b75f734 2023-04-12 15:54:14,540 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=128 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/B/65348727ec61498aa81911cb51be9bd1 2023-04-12 15:54:14,557 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. as already flushing 2023-04-12 15:54:14,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on ca92eda5f752c4c8f5e2f01f66bc73d8 2023-04-12 15:54:14,574 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=128 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/C/71bc051ce19a4e018b6c2e31c6d28c04 2023-04-12 15:54:14,580 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/A/374549cd7be54c8ca75b1c947b75f734 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/374549cd7be54c8ca75b1c947b75f734 2023-04-12 15:54:14,581 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:14,581 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:14,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34122 deadline: 1681314914572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:14,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34134 deadline: 1681314914573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:14,584 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/374549cd7be54c8ca75b1c947b75f734, entries=150, sequenceid=128, filesize=25.1 K 2023-04-12 15:54:14,585 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:14,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34094 deadline: 1681314914581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:14,585 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:14,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34066 deadline: 1681314914581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:14,586 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/B/65348727ec61498aa81911cb51be9bd1 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/65348727ec61498aa81911cb51be9bd1 2023-04-12 15:54:14,591 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/65348727ec61498aa81911cb51be9bd1, entries=150, sequenceid=128, filesize=11.6 K 2023-04-12 15:54:14,592 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/C/71bc051ce19a4e018b6c2e31c6d28c04 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/71bc051ce19a4e018b6c2e31c6d28c04 2023-04-12 15:54:14,598 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/71bc051ce19a4e018b6c2e31c6d28c04, entries=150, sequenceid=128, filesize=11.6 K 2023-04-12 15:54:14,599 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for ca92eda5f752c4c8f5e2f01f66bc73d8 in 123ms, sequenceid=128, compaction requested=true 2023-04-12 15:54:14,600 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ca92eda5f752c4c8f5e2f01f66bc73d8: 2023-04-12 15:54:14,600 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:14,600 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:14,600 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:14,600 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:14,600 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:14,600 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:14,602 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:14,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:14,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:14,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:14,602 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:14,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:14,603 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:14,603 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:14,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:14,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:14,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:14,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:14,605 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:14,605 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:14,605 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@241f2442[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:14,605 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:14,606 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:14,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:14,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:14,607 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:14,607 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:14,607 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:14,607 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:14,607 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:14,609 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:14,609 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:14,609 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:14,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:14,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:14,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:14,609 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:14,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:14,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:14,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:14,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:14,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:14,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:14,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:14,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:14,612 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:14,612 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:14,612 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:14,612 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:14,612 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:14,612 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:14,614 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:14,614 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:14,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:14,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:14,614 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:14,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:14,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:14,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:14,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:14,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:14,615 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:14,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:14,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:14,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:14,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:14,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:14,616 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:14,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:14,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:14,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:14,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:14,618 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:14,618 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:14,619 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:14,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:14,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:14,619 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:14,619 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:14,619 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:14,619 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:14,619 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:14,619 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:14,619 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:14,619 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:14,619 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:14,619 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:14,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:14,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:14,619 DEBUG [Thread-831] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:14,619 DEBUG [Thread-831] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:14,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on ca92eda5f752c4c8f5e2f01f66bc73d8 2023-04-12 15:54:14,685 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ca92eda5f752c4c8f5e2f01f66bc73d8 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-12 15:54:14,685 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681314854572, max timestamp=1681314854683Num uniques 859; 2023-04-12 15:54:14,685 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-12 15:54:14,686 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681314854572, max timestamp=1681314854683Num uniques 859; 2023-04-12 15:54:14,686 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-12 15:54:14,686 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681314854572, max timestamp=1681314854683Num uniques 859; 2023-04-12 15:54:14,686 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-12 15:54:14,687 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca92eda5f752c4c8f5e2f01f66bc73d8, store=A 2023-04-12 15:54:14,687 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:54:14,687 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca92eda5f752c4c8f5e2f01f66bc73d8, store=B 2023-04-12 15:54:14,687 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:54:14,687 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca92eda5f752c4c8f5e2f01f66bc73d8, store=C 2023-04-12 15:54:14,687 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:54:14,705 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230412adfe26944f2f4de2bc7e1c70d54f5641 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412adfe26944f2f4de2bc7e1c70d54f5641 2023-04-12 15:54:14,714 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681314854705, max timestamp=1681314854712Num uniques 859; 2023-04-12 15:54:14,714 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-12 15:54:14,714 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681314854705, max timestamp=1681314854712Num uniques 859; 2023-04-12 15:54:14,715 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-12 15:54:14,715 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681314854708, max timestamp=1681314854712Num uniques 859; 2023-04-12 15:54:14,715 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-12 15:54:14,720 DEBUG [Thread-831] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:14,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:14,726 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:14,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:14,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:14,729 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:14,729 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:14,729 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:14,729 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:14,729 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:14,729 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:14,731 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:14,731 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:14,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:14,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:14,731 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:14,731 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:14,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:14,732 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:14,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:14,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:14,733 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:14,734 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:14,734 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:14,734 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:14,734 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:14,734 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:14,735 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:14,735 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:14,735 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:14,735 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. started... 2023-04-12 15:54:14,735 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. as already flushing 2023-04-12 15:54:14,735 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:14,735 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:14,735 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:14,735 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:14,735 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:14,735 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:14,738 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:14,738 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:14,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:14,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:14,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:14,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:14,738 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:14,738 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:14,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:14,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:14,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:14,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:14,741 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:14,741 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b877fef[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:14,741 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:14,741 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:14,743 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:14,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:14,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:14,743 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:14,743 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:14,743 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:14,744 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:14,745 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:14,745 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:14,745 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:14,746 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:14,746 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:14,746 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:14,746 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:14,746 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:14,746 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:14,746 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:14,748 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:14,748 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:14,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:14,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:14,748 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:14,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:14,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:14,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:14,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:14,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:14,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:14,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:14,749 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:14,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:14,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:14,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:14,751 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:14,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:14,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:14,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:14,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:14,754 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=418, cellSize=19138, totalHeapSize=43712, min timestamp=1681314854728, max timestamp=1681314854751Num uniques 1009; 2023-04-12 15:54:14,754 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=19138, suffix heap size=132904,new segment heap size=43712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=418 2023-04-12 15:54:14,755 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=418, cellSize=19138, totalHeapSize=43712, min timestamp=1681314854728, max timestamp=1681314854751Num uniques 1068; 2023-04-12 15:54:14,755 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=418, cellSize=19138, totalHeapSize=43712, min timestamp=1681314854728, max timestamp=1681314854751Num uniques 1068; 2023-04-12 15:54:14,755 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=19138, suffix heap size=139040,new segment heap size=43712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=418 2023-04-12 15:54:14,755 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:14,755 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=19138, suffix heap size=139040,new segment heap size=43712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=418 2023-04-12 15:54:14,755 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:14,755 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:14,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:14,755 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:14,755 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:14,755 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:14,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:14,755 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:14,755 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:14,755 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:14,755 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:14,755 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:14,755 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:14,756 DEBUG [Thread-831] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:14,756 DEBUG [Thread-831] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:14,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:14,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:14,756 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:14,787 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:14,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34066 deadline: 1681314914782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:14,792 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:14,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34134 deadline: 1681314914783, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:14,793 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:14,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34122 deadline: 1681314914784, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:14,793 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:14,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34094 deadline: 1681314914785, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:14,856 DEBUG [Thread-831] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:14,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:14,858 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:14,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:14,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:14,860 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:14,860 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:14,860 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:14,860 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:14,860 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:14,860 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:14,862 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:14,862 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:14,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:14,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:14,862 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:14,862 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:14,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:14,862 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:14,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:14,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:14,863 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:14,863 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:14,863 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:14,863 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:14,863 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:14,863 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:14,864 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:14,864 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:14,864 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. started... 2023-04-12 15:54:14,864 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:14,864 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. as already flushing 2023-04-12 15:54:14,864 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:14,864 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:14,864 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:14,864 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:14,864 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:14,864 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:14,866 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:14,866 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:14,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:14,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:14,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:14,866 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:14,866 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:14,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:14,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:14,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:14,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:14,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:14,868 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:14,868 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b0b36b7[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:14,868 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:14,868 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:14,869 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:14,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:14,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:14,869 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:14,869 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:14,869 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:14,869 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:14,869 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:14,874 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:14,874 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:14,874 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:14,874 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:14,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:14,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:14,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:14,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:14,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:14,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:14,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:14,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:14,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:14,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:14,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:14,877 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:14,877 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:14,877 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:14,877 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:14,877 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:14,877 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:14,879 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:14,879 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:14,879 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:14,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:14,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:14,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:14,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:14,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:14,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:14,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:14,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:14,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:14,880 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:14,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:14,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:14,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:14,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:14,881 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:14,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:14,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:14,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:14,884 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:14,884 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:14,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:14,884 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:14,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:14,884 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:14,884 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:14,884 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:14,884 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:14,884 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:14,884 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:14,884 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:14,884 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:14,884 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:14,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:14,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:14,885 DEBUG [Thread-831] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:14,885 DEBUG [Thread-831] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:14,891 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:14,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34066 deadline: 1681314914889, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:14,897 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:14,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34122 deadline: 1681314914894, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:14,898 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:14,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34134 deadline: 1681314914895, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:14,898 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:14,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34094 deadline: 1681314914895, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:14,985 DEBUG [Thread-831] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:14,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:14,987 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:14,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:14,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:14,992 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:14,992 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:14,992 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:14,992 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:14,993 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:14,993 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:14,994 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:14,994 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:14,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:14,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:14,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:14,994 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:14,995 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:14,995 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:14,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:14,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:14,995 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:15,001 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:15,001 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:15,001 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:15,001 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:15,001 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:15,001 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:15,001 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:15,001 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:15,001 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. started... 2023-04-12 15:54:15,002 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. as already flushing 2023-04-12 15:54:15,002 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:15,002 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:15,002 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:15,002 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:15,002 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:15,002 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:15,004 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:15,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:15,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:15,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:15,005 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:15,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:15,005 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:15,005 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:15,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:15,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:15,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:15,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:15,007 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:15,007 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22365360[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:15,007 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:15,007 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:15,008 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:15,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:15,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:15,008 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:15,009 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:15,008 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:15,009 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:15,009 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:15,010 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:15,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:15,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:15,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:15,010 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:15,010 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:15,010 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:15,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:15,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:15,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:15,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:15,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:15,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:15,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:15,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:15,013 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:15,013 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:15,013 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:15,013 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:15,013 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:15,013 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:15,015 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:15,015 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:15,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:15,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:15,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:15,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:15,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:15,015 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:15,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:15,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:15,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:15,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:15,017 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:15,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:15,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:15,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:15,018 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:15,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:15,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:15,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:15,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:15,022 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:15,022 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:15,022 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:15,022 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:15,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:15,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:15,022 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:15,023 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:15,023 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:15,023 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:15,023 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:15,023 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:15,023 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:15,023 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:15,023 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:15,023 DEBUG [Thread-831] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:15,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:15,023 DEBUG [Thread-831] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:15,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:15,097 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:15,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34066 deadline: 1681314915092, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:15,099 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:15,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34134 deadline: 1681314915099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:15,102 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:15,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34122 deadline: 1681314915100, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:15,103 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:15,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34094 deadline: 1681314915103, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:15,117 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=150, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/A/94e672eeb69c414ea6c42bd1b8d07eb7 2023-04-12 15:54:15,123 DEBUG [Thread-831] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:15,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:15,128 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:15,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:15,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:15,130 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:15,130 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:15,130 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:15,130 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:15,131 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:15,131 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:15,132 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:15,132 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:15,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:15,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:15,133 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:15,133 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:15,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:15,133 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:15,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:15,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:15,134 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:15,140 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:15,140 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:15,140 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:15,140 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:15,140 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:15,141 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:15,141 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:15,143 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:15,143 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. started... 2023-04-12 15:54:15,143 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. as already flushing 2023-04-12 15:54:15,143 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:15,143 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:15,143 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:15,143 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:15,143 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:15,143 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:15,145 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:15,145 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:15,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:15,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:15,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:15,145 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:15,145 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:15,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:15,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:15,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:15,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:15,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:15,147 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:15,147 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f565e50[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:15,147 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:15,147 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:15,149 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:15,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:15,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:15,149 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:15,149 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:15,149 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:15,149 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:15,149 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:15,151 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:15,151 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:15,151 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:15,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:15,151 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:15,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:15,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:15,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:15,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:15,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:15,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:15,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:15,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:15,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:15,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:15,155 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:15,155 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:15,155 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:15,155 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:15,155 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:15,155 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:15,157 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:15,157 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:15,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:15,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:15,157 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:15,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:15,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:15,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:15,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:15,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:15,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:15,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:15,158 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:15,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:15,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:15,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:15,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:15,161 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:15,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:15,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:15,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:15,163 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:15,163 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:15,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:15,163 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:15,163 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:15,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:15,163 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:15,163 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:15,163 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:15,164 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:15,164 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:15,164 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:15,164 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:15,164 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:15,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:15,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:15,164 DEBUG [Thread-831] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:15,164 DEBUG [Thread-831] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:15,264 DEBUG [Thread-831] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:15,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:15,269 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:15,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:15,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:15,276 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:15,276 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:15,276 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:15,276 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:15,276 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:15,276 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:15,277 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:15,277 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:15,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:15,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:15,278 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:15,278 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:15,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:15,278 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:15,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:15,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:15,279 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:15,280 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:15,280 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:15,280 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:15,280 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:15,280 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:15,288 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:15,288 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:15,288 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:15,288 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. started... 2023-04-12 15:54:15,288 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. as already flushing 2023-04-12 15:54:15,288 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:15,289 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:15,289 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:15,289 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:15,289 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:15,289 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:15,291 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:15,291 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:15,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:15,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:15,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:15,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:15,292 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:15,292 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:15,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:15,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:15,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:15,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:15,293 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:15,294 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@fde2e52[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:15,294 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:15,294 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:15,295 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:15,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:15,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:15,295 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:15,295 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:15,295 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:15,295 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:15,295 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:15,296 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:15,296 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:15,296 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:15,296 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:15,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:15,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:15,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:15,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:15,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:15,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:15,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:15,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:15,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:15,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:15,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:15,300 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:15,300 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:15,300 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:15,300 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:15,300 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:15,300 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:15,305 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:15,305 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:15,305 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:15,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:15,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:15,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:15,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:15,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:15,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:15,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:15,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:15,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:15,307 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:15,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:15,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:15,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:15,311 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:15,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:15,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:15,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:15,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:15,314 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:15,314 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:15,314 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:15,314 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:15,314 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:15,314 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:15,314 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:15,314 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:15,314 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:15,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:15,314 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:15,314 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:15,314 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:15,314 DEBUG [Thread-831] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:15,315 DEBUG [Thread-831] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:15,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:15,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:15,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:15,315 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:15,406 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:15,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34066 deadline: 1681314915400, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:15,407 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:15,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34134 deadline: 1681314915401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:15,408 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:15,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34122 deadline: 1681314915405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:15,408 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:15,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34094 deadline: 1681314915406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:15,415 DEBUG [Thread-831] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:15,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:15,417 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:15,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:15,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:15,418 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:15,418 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:15,418 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:15,418 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:15,419 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:15,419 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:15,420 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:15,420 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:15,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:15,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:15,420 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:15,420 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:15,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:15,420 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:15,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:15,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:15,421 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:15,421 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:15,421 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:15,421 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:15,421 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:15,421 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:15,421 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:15,421 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:15,421 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:15,421 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. started... 2023-04-12 15:54:15,422 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. as already flushing 2023-04-12 15:54:15,422 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:15,422 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:15,422 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:15,422 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:15,422 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:15,422 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:15,423 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:15,423 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:15,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:15,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:15,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:15,423 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:15,423 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:15,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:15,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:15,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:15,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:15,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:15,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:15,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d320f90[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:15,425 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:15,425 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:15,427 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:15,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:15,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:15,427 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:15,427 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:15,427 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:15,427 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:15,427 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:15,428 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:15,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:15,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:15,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:15,428 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:15,429 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:15,429 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:15,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:15,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:15,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:15,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:15,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:15,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:15,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:15,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:15,431 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:15,431 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:15,431 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:15,431 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:15,432 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:15,432 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:15,433 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:15,433 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:15,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:15,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:15,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:15,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:15,434 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:15,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:15,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:15,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:15,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:15,434 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:15,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:15,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:15,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:15,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:15,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:15,435 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:15,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:15,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:15,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:15,437 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:15,437 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:15,437 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:15,438 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:15,438 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:15,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:15,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:15,438 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:15,438 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:15,438 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:15,438 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:15,438 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:15,438 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:15,438 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:15,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:15,438 DEBUG [Thread-831] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:15,438 DEBUG [Thread-831] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:15,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:15,538 DEBUG [Thread-831] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:15,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:15,540 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:15,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:15,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:15,542 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:15,542 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:15,542 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:15,542 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:15,542 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:15,542 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:15,546 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=150 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/B/5691f3347f7b4b9190772503c31e3a70 2023-04-12 15:54:15,546 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:15,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:15,546 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:15,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:15,546 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:15,546 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:15,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:15,547 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:15,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:15,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:15,547 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:15,548 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:15,548 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:15,548 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:15,548 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:15,548 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:15,549 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:15,549 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:15,549 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:15,549 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. started... 2023-04-12 15:54:15,549 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. as already flushing 2023-04-12 15:54:15,549 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:15,549 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:15,549 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:15,549 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:15,549 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:15,549 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:15,551 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:15,551 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:15,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:15,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:15,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:15,551 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:15,551 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:15,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:15,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:15,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:15,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:15,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:15,553 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:15,553 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d0fea05[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:15,553 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:15,553 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:15,554 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:15,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:15,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:15,554 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:15,555 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:15,555 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:15,555 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:15,555 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:15,556 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:15,556 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:15,556 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:15,556 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:15,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:15,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:15,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:15,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:15,557 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=150 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/C/e46783633842420aba5c9edfe4553aca 2023-04-12 15:54:15,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:15,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:15,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:15,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:15,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:15,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:15,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:15,561 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:15,561 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:15,561 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:15,561 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:15,561 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:15,561 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:15,562 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/A/94e672eeb69c414ea6c42bd1b8d07eb7 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/94e672eeb69c414ea6c42bd1b8d07eb7 2023-04-12 15:54:15,564 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:15,564 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:15,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:15,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:15,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:15,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:15,564 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:15,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:15,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:15,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:15,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:15,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:15,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:15,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:15,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:15,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:15,567 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/94e672eeb69c414ea6c42bd1b8d07eb7, entries=250, sequenceid=150, filesize=38.9 K 2023-04-12 15:54:15,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:15,568 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/B/5691f3347f7b4b9190772503c31e3a70 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/5691f3347f7b4b9190772503c31e3a70 2023-04-12 15:54:15,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:15,569 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:15,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:15,571 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:15,574 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/5691f3347f7b4b9190772503c31e3a70, entries=150, sequenceid=150, filesize=11.8 K 2023-04-12 15:54:15,574 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:15,574 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:15,574 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:15,574 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:15,574 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:15,574 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:15,574 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:15,574 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:15,574 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:15,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:15,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:15,575 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:15,575 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:15,575 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:15,575 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/C/e46783633842420aba5c9edfe4553aca as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/e46783633842420aba5c9edfe4553aca 2023-04-12 15:54:15,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:15,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:15,575 DEBUG [Thread-831] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:15,575 DEBUG [Thread-831] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:15,582 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/e46783633842420aba5c9edfe4553aca, entries=150, sequenceid=150, filesize=11.8 K 2023-04-12 15:54:15,583 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~86.11 KB/88176, currentSize=161.02 KB/164880 for ca92eda5f752c4c8f5e2f01f66bc73d8 in 898ms, sequenceid=150, compaction requested=true 2023-04-12 15:54:15,583 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ca92eda5f752c4c8f5e2f01f66bc73d8: 2023-04-12 15:54:15,583 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:54:15,583 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-12 15:54:15,584 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:54:15,584 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-12 15:54:15,584 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:54:15,586 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 168713 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-12 15:54:15,586 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1912): ca92eda5f752c4c8f5e2f01f66bc73d8/A is initiating minor compaction (all files) 2023-04-12 15:54:15,586 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71569 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-12 15:54:15,586 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ca92eda5f752c4c8f5e2f01f66bc73d8/A in TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:15,586 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1912): ca92eda5f752c4c8f5e2f01f66bc73d8/B is initiating minor compaction (all files) 2023-04-12 15:54:15,586 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/8d3fbd974ffd4340b479f0bcf22e0b01, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/5abb434e890b43f8b48a8dfc06182063, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/da539482f9214cc69b85aaee0c00fecc, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/62b43f9b9d574e0882527b21f5e022cc, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/374549cd7be54c8ca75b1c947b75f734, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/94e672eeb69c414ea6c42bd1b8d07eb7] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp, totalSize=164.8 K 2023-04-12 15:54:15,586 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ca92eda5f752c4c8f5e2f01f66bc73d8/B in TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:15,586 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/b6eb09dc371b4a5fa98c56978e88aaa0, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/9f1c46d5ce854a2b88438c74d405d1e7, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/841a0cf8efa64001929f1be245bbdbb0, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/4bae1d6cd92f4824844d04d8edbd5aa6, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/65348727ec61498aa81911cb51be9bd1, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/5691f3347f7b4b9190772503c31e3a70] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp, totalSize=69.9 K 2023-04-12 15:54:15,587 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 8d3fbd974ffd4340b479f0bcf22e0b01, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1681314850422 2023-04-12 15:54:15,587 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting b6eb09dc371b4a5fa98c56978e88aaa0, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1681314850422 2023-04-12 15:54:15,587 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 5abb434e890b43f8b48a8dfc06182063, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1681314851046 2023-04-12 15:54:15,587 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 9f1c46d5ce854a2b88438c74d405d1e7, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1681314851046 2023-04-12 15:54:15,588 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting da539482f9214cc69b85aaee0c00fecc, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=96, earliestPutTs=1681314851672 2023-04-12 15:54:15,588 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 841a0cf8efa64001929f1be245bbdbb0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=96, earliestPutTs=1681314851672 2023-04-12 15:54:15,588 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 62b43f9b9d574e0882527b21f5e022cc, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=116, earliestPutTs=1681314852805 2023-04-12 15:54:15,588 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 4bae1d6cd92f4824844d04d8edbd5aa6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=116, earliestPutTs=1681314852805 2023-04-12 15:54:15,588 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 374549cd7be54c8ca75b1c947b75f734, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=128, earliestPutTs=1681314853933 2023-04-12 15:54:15,589 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 65348727ec61498aa81911cb51be9bd1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=128, earliestPutTs=1681314853933 2023-04-12 15:54:15,589 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 94e672eeb69c414ea6c42bd1b8d07eb7, keycount=250, bloomtype=ROW, size=38.9 K, encoding=NONE, compression=NONE, seqNum=150, earliestPutTs=1681314854572 2023-04-12 15:54:15,589 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 5691f3347f7b4b9190772503c31e3a70, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=150, earliestPutTs=1681314854576 2023-04-12 15:54:15,602 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] throttle.PressureAwareThroughputController(145): ca92eda5f752c4c8f5e2f01f66bc73d8#B#compaction#130 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:54:15,614 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] throttle.PressureAwareThroughputController(145): ca92eda5f752c4c8f5e2f01f66bc73d8#A#compaction#129 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:54:15,675 DEBUG [Thread-831] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:15,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:15,677 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:15,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:15,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:15,679 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:15,679 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:15,679 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:15,679 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:15,679 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:15,680 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:15,681 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:15,681 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:15,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:15,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:15,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:15,681 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:15,681 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:15,682 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:15,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:15,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:15,682 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:15,682 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:15,682 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:15,682 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:15,682 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:15,682 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:15,683 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:15,683 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:15,683 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:15,683 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. started... 2023-04-12 15:54:15,683 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ca92eda5f752c4c8f5e2f01f66bc73d8 3/3 column families, dataSize=161.02 KB heapSize=440.39 KB 2023-04-12 15:54:15,683 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca92eda5f752c4c8f5e2f01f66bc73d8, store=A 2023-04-12 15:54:15,683 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:54:15,683 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca92eda5f752c4c8f5e2f01f66bc73d8, store=B 2023-04-12 15:54:15,683 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:54:15,683 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca92eda5f752c4c8f5e2f01f66bc73d8, store=C 2023-04-12 15:54:15,683 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:54:15,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on ca92eda5f752c4c8f5e2f01f66bc73d8 2023-04-12 15:54:15,913 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. as already flushing 2023-04-12 15:54:15,921 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:15,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34122 deadline: 1681314915920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:15,921 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:15,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34066 deadline: 1681314915920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:15,924 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:15,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34094 deadline: 1681314915921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:15,924 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:15,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34134 deadline: 1681314915921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:16,023 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:16,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34066 deadline: 1681314916022, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:16,023 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:16,023 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/B/83437973030343d19c239e466a565a4c as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/83437973030343d19c239e466a565a4c 2023-04-12 15:54:16,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34122 deadline: 1681314916023, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:16,031 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:16,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34094 deadline: 1681314916025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:16,031 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:16,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34134 deadline: 1681314916025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:16,034 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in ca92eda5f752c4c8f5e2f01f66bc73d8/B of ca92eda5f752c4c8f5e2f01f66bc73d8 into 83437973030343d19c239e466a565a4c(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:54:16,034 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ca92eda5f752c4c8f5e2f01f66bc73d8: 2023-04-12 15:54:16,034 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8., storeName=ca92eda5f752c4c8f5e2f01f66bc73d8/B, priority=10, startTime=1681314855584; duration=0sec 2023-04-12 15:54:16,034 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:54:16,034 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-12 15:54:16,036 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71569 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-12 15:54:16,037 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1912): ca92eda5f752c4c8f5e2f01f66bc73d8/C is initiating minor compaction (all files) 2023-04-12 15:54:16,037 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ca92eda5f752c4c8f5e2f01f66bc73d8/C in TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:16,037 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/e0f9f5ef76834f83a88df39bfdf88372, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/4ed8ee44d96b4498869b3612f17a7985, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/93baa4de2d5042c2a2bd03d3ea36df9c, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/5611fc74272649b39d04a33911779ca3, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/71bc051ce19a4e018b6c2e31c6d28c04, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/e46783633842420aba5c9edfe4553aca] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp, totalSize=69.9 K 2023-04-12 15:54:16,037 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting e0f9f5ef76834f83a88df39bfdf88372, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1681314850422 2023-04-12 15:54:16,037 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 4ed8ee44d96b4498869b3612f17a7985, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1681314851046 2023-04-12 15:54:16,038 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 93baa4de2d5042c2a2bd03d3ea36df9c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=96, earliestPutTs=1681314851672 2023-04-12 15:54:16,038 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 5611fc74272649b39d04a33911779ca3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=116, earliestPutTs=1681314852805 2023-04-12 15:54:16,040 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 71bc051ce19a4e018b6c2e31c6d28c04, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=128, earliestPutTs=1681314853933 2023-04-12 15:54:16,040 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting e46783633842420aba5c9edfe4553aca, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=150, earliestPutTs=1681314854576 2023-04-12 15:54:16,050 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] throttle.PressureAwareThroughputController(145): ca92eda5f752c4c8f5e2f01f66bc73d8#C#compaction#132 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:54:16,067 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/C/eaaeb56b20b14f5fb63fc8680ac76872 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/eaaeb56b20b14f5fb63fc8680ac76872 2023-04-12 15:54:16,075 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in ca92eda5f752c4c8f5e2f01f66bc73d8/C of ca92eda5f752c4c8f5e2f01f66bc73d8 into eaaeb56b20b14f5fb63fc8680ac76872(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:54:16,075 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ca92eda5f752c4c8f5e2f01f66bc73d8: 2023-04-12 15:54:16,075 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8., storeName=ca92eda5f752c4c8f5e2f01f66bc73d8/C, priority=10, startTime=1681314855584; duration=0sec 2023-04-12 15:54:16,075 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:54:16,105 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304121bee7472e58b4912a993c578954f0739 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304121bee7472e58b4912a993c578954f0739 2023-04-12 15:54:16,110 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=203, memsize=53.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/A/5c76f100bfc040a9b61197c36c380c46 2023-04-12 15:54:16,228 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:16,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34066 deadline: 1681314916224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:16,228 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:16,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34122 deadline: 1681314916225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:16,234 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:16,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34094 deadline: 1681314916233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:16,235 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:16,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34134 deadline: 1681314916233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:16,523 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=203 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/B/d1413f20510a46dc9d1d84e13ecb0083 2023-04-12 15:54:16,530 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:16,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34066 deadline: 1681314916529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:16,536 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:16,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34122 deadline: 1681314916531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:16,553 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:16,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34134 deadline: 1681314916551, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:16,553 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:16,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34094 deadline: 1681314916551, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:16,846 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/A/7ff490ee87994b168395323b44b276ec as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/7ff490ee87994b168395323b44b276ec 2023-04-12 15:54:16,851 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in ca92eda5f752c4c8f5e2f01f66bc73d8/A of ca92eda5f752c4c8f5e2f01f66bc73d8 into 7ff490ee87994b168395323b44b276ec(size=25.6 K), total size for store is 25.6 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-12 15:54:16,851 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ca92eda5f752c4c8f5e2f01f66bc73d8: 2023-04-12 15:54:16,852 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8., storeName=ca92eda5f752c4c8f5e2f01f66bc73d8/A, priority=10, startTime=1681314855583; duration=1sec 2023-04-12 15:54:16,852 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:54:16,935 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=203 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/C/c6bef8e944eb405aa018c1fee08f16d6 2023-04-12 15:54:16,940 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/A/5c76f100bfc040a9b61197c36c380c46 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/5c76f100bfc040a9b61197c36c380c46 2023-04-12 15:54:16,945 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/5c76f100bfc040a9b61197c36c380c46, entries=150, sequenceid=203, filesize=25.3 K 2023-04-12 15:54:16,945 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/B/d1413f20510a46dc9d1d84e13ecb0083 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/d1413f20510a46dc9d1d84e13ecb0083 2023-04-12 15:54:16,950 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/d1413f20510a46dc9d1d84e13ecb0083, entries=150, sequenceid=203, filesize=11.8 K 2023-04-12 15:54:16,950 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/C/c6bef8e944eb405aa018c1fee08f16d6 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/c6bef8e944eb405aa018c1fee08f16d6 2023-04-12 15:54:16,954 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/c6bef8e944eb405aa018c1fee08f16d6, entries=150, sequenceid=203, filesize=11.8 K 2023-04-12 15:54:16,955 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~161.02 KB/164880, heapSize ~440.34 KB/450912, currentSize=33.54 KB/34350 for ca92eda5f752c4c8f5e2f01f66bc73d8 in 1272ms, sequenceid=203, compaction requested=false 2023-04-12 15:54:16,955 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ca92eda5f752c4c8f5e2f01f66bc73d8: 2023-04-12 15:54:16,955 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:16,955 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:16,955 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:16,955 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:16,955 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:16,955 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:16,958 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:16,958 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:16,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:16,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:16,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:16,958 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:16,958 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:16,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:16,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:16,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:16,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:16,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:16,960 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:16,960 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66992a[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:16,960 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:16,960 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:16,963 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:16,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:16,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:16,963 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:16,963 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:16,963 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:16,963 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:16,963 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:16,964 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:16,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:16,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:16,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:16,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:16,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:16,965 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:16,965 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:16,965 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:16,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:16,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:16,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:16,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:16,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:16,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:16,967 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:16,967 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:16,967 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:16,967 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:16,967 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:16,967 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:16,968 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:16,968 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:16,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:16,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:16,968 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:16,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:16,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:16,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:16,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:16,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:16,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:16,969 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:16,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:16,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:16,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:16,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:16,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:16,970 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:16,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:16,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:16,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:16,976 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:16,976 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:16,976 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:16,976 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:16,976 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:16,976 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:16,976 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:16,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:16,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:16,976 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:16,976 DEBUG [Thread-831] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:16,976 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:16,976 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:16,976 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:16,976 DEBUG [Thread-831] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:16,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:16,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:16,976 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:17,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on ca92eda5f752c4c8f5e2f01f66bc73d8 2023-04-12 15:54:17,035 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ca92eda5f752c4c8f5e2f01f66bc73d8 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-12 15:54:17,035 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca92eda5f752c4c8f5e2f01f66bc73d8, store=A 2023-04-12 15:54:17,035 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:17,035 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca92eda5f752c4c8f5e2f01f66bc73d8, store=B 2023-04-12 15:54:17,035 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:17,035 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca92eda5f752c4c8f5e2f01f66bc73d8, store=C 2023-04-12 15:54:17,035 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:17,049 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041257d1a2a7b24b4e3ab01fda16c035537e to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041257d1a2a7b24b4e3ab01fda16c035537e 2023-04-12 15:54:17,068 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681314857059, max timestamp=1681314857064Num uniques 859; 2023-04-12 15:54:17,068 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681314857059, max timestamp=1681314857064Num uniques 859; 2023-04-12 15:54:17,068 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-12 15:54:17,068 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-12 15:54:17,068 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681314857059, max timestamp=1681314857064Num uniques 859; 2023-04-12 15:54:17,068 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-12 15:54:17,077 DEBUG [Thread-831] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:17,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:17,079 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:17,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:17,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:17,080 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:17,080 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:17,080 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:17,080 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:17,080 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:17,080 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:17,082 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,082 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:17,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:17,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:17,082 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,082 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:17,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:17,082 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:17,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:17,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:17,083 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:17,083 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:17,083 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:17,083 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:17,083 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:17,083 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:17,083 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:17,083 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:17,083 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:17,084 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. started... 2023-04-12 15:54:17,084 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. as already flushing 2023-04-12 15:54:17,084 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:17,084 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:17,084 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:17,084 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,084 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:17,084 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:17,085 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:17,085 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:17,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:17,086 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:17,086 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:17,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:17,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:17,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:17,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:17,088 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:17,088 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7eb49bca[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:17,088 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:17,088 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:17,089 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:17,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:17,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:17,089 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:17,089 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,090 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:17,089 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:17,090 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:17,091 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:17,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34134 deadline: 1681314917088, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,091 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:17,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34066 deadline: 1681314917088, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,091 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:17,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34094 deadline: 1681314917089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,091 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:17,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34122 deadline: 1681314917089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,092 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:17,092 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:17,092 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:17,092 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:17,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:17,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:17,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:17,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:17,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:17,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:17,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:17,095 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:17,095 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:17,095 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:17,095 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:17,095 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:17,095 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:17,097 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:17,097 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:17,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:17,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:17,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:17,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:17,097 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:17,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:17,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:17,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:17,097 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:17,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:17,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:17,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:17,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:17,099 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:17,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:17,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,101 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,101 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:17,101 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:17,101 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:17,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:17,101 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:17,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:17,101 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:17,101 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:17,101 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,101 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:17,101 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:17,101 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:17,101 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:17,101 DEBUG [Thread-831] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:17,101 DEBUG [Thread-831] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:17,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:17,102 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:17,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:17,192 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:17,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34134 deadline: 1681314917192, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,193 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:17,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34094 deadline: 1681314917192, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,193 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:17,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34066 deadline: 1681314917192, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,198 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:17,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34122 deadline: 1681314917194, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,202 DEBUG [Thread-831] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:17,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:17,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:17,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:17,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:17,205 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:17,205 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:17,205 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:17,205 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:17,205 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:17,205 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:17,207 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:17,207 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:17,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:17,207 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,207 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:17,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:17,208 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:17,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:17,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:17,208 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:17,208 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:17,208 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:17,208 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:17,209 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:17,209 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:17,209 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:17,209 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:17,209 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:17,209 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. started... 2023-04-12 15:54:17,209 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. as already flushing 2023-04-12 15:54:17,209 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:17,209 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:17,209 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:17,209 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,209 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:17,209 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:17,212 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,212 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:17,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:17,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:17,212 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:17,212 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:17,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:17,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:17,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:17,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:17,214 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:17,215 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35c1b86a[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:17,215 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:17,215 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:17,216 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:17,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:17,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:17,216 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:17,216 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:17,216 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:17,217 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,217 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:17,221 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:17,221 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:17,221 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,221 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:17,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:17,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:17,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:17,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:17,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:17,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:17,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:17,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:17,224 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:17,224 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:17,224 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:17,224 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:17,224 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:17,224 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:17,226 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:17,226 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:17,226 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:17,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:17,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:17,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:17,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:17,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:17,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:17,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:17,227 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:17,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:17,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:17,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:17,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:17,229 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:17,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:17,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,233 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,233 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:17,233 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:17,233 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,233 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:17,233 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:17,233 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:17,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:17,233 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:17,233 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:17,233 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:17,234 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:17,233 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:17,234 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:17,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:17,234 DEBUG [Thread-831] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:17,234 DEBUG [Thread-831] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:17,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:17,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:17,334 DEBUG [Thread-831] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:17,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:17,336 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:17,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:17,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:17,337 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:17,337 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:17,337 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:17,338 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:17,338 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:17,338 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:17,339 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,339 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:17,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:17,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:17,339 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,340 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:17,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:17,340 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:17,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:17,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:17,340 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:17,341 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:17,341 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:17,341 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:17,341 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:17,341 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:17,341 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:17,341 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:17,341 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:17,341 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. started... 2023-04-12 15:54:17,341 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. as already flushing 2023-04-12 15:54:17,341 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:17,341 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:17,341 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:17,341 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,342 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:17,342 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:17,344 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,344 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:17,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:17,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:17,345 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:17,345 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:17,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:17,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:17,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:17,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:17,346 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:17,346 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ff11fd3[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:17,346 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:17,346 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:17,347 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:17,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:17,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:17,347 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:17,347 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:17,347 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:17,348 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,348 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:17,350 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:17,350 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:17,350 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,350 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:17,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:17,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:17,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:17,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:17,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:17,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:17,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:17,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:17,353 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:17,353 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:17,353 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:17,353 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:17,353 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:17,353 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:17,355 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:17,355 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:17,355 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:17,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:17,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:17,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:17,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:17,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:17,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:17,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:17,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:17,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:17,356 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:17,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:17,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:17,357 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:17,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:17,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,362 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,362 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:17,362 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:17,362 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:17,362 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:17,362 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:17,363 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:17,363 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:17,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:17,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:17,363 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,363 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:17,363 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:17,363 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:17,363 DEBUG [Thread-831] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:17,363 DEBUG [Thread-831] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:17,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:17,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:17,363 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:17,398 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:17,398 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:17,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34134 deadline: 1681314917394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34066 deadline: 1681314917394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,399 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:17,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34094 deadline: 1681314917395, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,401 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:17,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34122 deadline: 1681314917400, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,455 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=216, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/A/3d54ed87c8084576afaf5a7e004de4ea 2023-04-12 15:54:17,463 DEBUG [Thread-831] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:17,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:17,466 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:17,467 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=216 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/B/9bf16651b56c467eb714602de7d72c09 2023-04-12 15:54:17,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:17,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:17,468 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:17,468 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:17,468 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:17,468 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:17,469 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:17,469 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:17,471 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:17,471 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:17,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:17,471 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,471 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:17,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:17,472 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:17,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:17,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:17,472 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:17,473 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:17,473 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:17,473 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:17,473 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:17,473 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:17,474 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:17,474 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:17,474 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:17,474 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. started... 2023-04-12 15:54:17,474 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. as already flushing 2023-04-12 15:54:17,474 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:17,474 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:17,474 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:17,474 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,474 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:17,474 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:17,476 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:17,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:17,476 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:17,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:17,476 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:17,476 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:17,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:17,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:17,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:17,478 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:17,478 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6667d073[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:17,478 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:17,478 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:17,479 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:17,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:17,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:17,479 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:17,479 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:17,479 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:17,479 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,479 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:17,479 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:17,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34118 deadline: 1681314917476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,484 DEBUG [Thread-829] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8200 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8., hostname=jenkins-hbase4.apache.org,38905,1681314818739, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-12 15:54:17,485 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,485 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:17,485 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:17,486 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:17,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:17,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:17,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:17,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:17,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:17,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:17,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:17,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:17,490 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:17,490 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:17,490 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:17,490 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:17,490 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:17,490 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:17,498 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:17,498 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:17,498 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:17,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:17,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:17,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:17,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:17,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:17,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:17,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:17,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:17,499 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:17,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:17,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:17,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:17,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,500 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:17,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:17,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,502 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,502 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:17,503 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:17,503 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,503 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:17,503 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:17,503 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:17,503 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:17,503 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:17,503 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:17,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:17,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:17,503 DEBUG [Thread-831] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:17,503 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:17,503 DEBUG [Thread-831] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:17,503 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:17,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:17,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:17,603 DEBUG [Thread-831] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:17,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:17,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:17,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:17,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:17,606 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:17,606 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:17,606 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:17,606 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:17,607 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:17,607 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:17,608 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:17,608 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:17,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:17,609 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,609 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:17,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:17,609 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:17,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:17,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:17,610 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:17,610 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:17,610 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:17,610 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:17,610 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:17,610 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:17,610 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:17,610 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:17,610 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:17,610 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. started... 2023-04-12 15:54:17,610 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. as already flushing 2023-04-12 15:54:17,611 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:17,611 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:17,611 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:17,611 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,611 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:17,611 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:17,612 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,612 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:17,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:17,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:17,612 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:17,612 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:17,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:17,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:17,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:17,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:17,614 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:17,614 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@319f2d53[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:17,614 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:17,614 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:17,616 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:17,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:17,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:17,616 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:17,616 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:17,616 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:17,616 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,616 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:17,617 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:17,617 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:17,617 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,618 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:17,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:17,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:17,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:17,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:17,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:17,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:17,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:17,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:17,620 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:17,620 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:17,620 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:17,620 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:17,620 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:17,620 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:17,624 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:17,624 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:17,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:17,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:17,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:17,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:17,624 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:17,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:17,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:17,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:17,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:17,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:17,625 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:17,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:17,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:17,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,626 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:17,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:17,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,629 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:17,629 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:17,629 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,629 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:17,629 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:17,629 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,629 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:17,629 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:17,629 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:17,629 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:17,633 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:17,633 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:17,634 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:17,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:17,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:17,634 DEBUG [Thread-831] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:17,634 DEBUG [Thread-831] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:17,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:17,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:17,701 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:17,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34066 deadline: 1681314917700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,702 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:17,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34094 deadline: 1681314917700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,702 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:17,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34134 deadline: 1681314917701, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,706 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:17,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34122 deadline: 1681314917704, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,734 DEBUG [Thread-831] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:17,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:17,736 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:17,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:17,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:17,737 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:17,738 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:17,738 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:17,738 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:17,738 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:17,738 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:17,740 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:17,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:17,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:17,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:17,741 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,742 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:17,742 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,742 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:17,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:17,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:17,743 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:17,743 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:17,743 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:17,743 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:17,743 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:17,743 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:17,743 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:17,743 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:17,743 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:17,743 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. started... 2023-04-12 15:54:17,744 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. as already flushing 2023-04-12 15:54:17,744 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:17,744 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:17,744 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:17,744 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,744 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:17,744 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:17,745 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:17,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:17,746 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:17,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:17,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:17,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:17,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,747 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:17,747 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:17,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:17,748 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:17,748 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9a5c1e2[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:17,748 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:17,749 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:17,750 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:17,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:17,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:17,750 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:17,750 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:17,750 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:17,751 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,751 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:17,752 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,752 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:17,752 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:17,752 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:17,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:17,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:17,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:17,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:17,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:17,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:17,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:17,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:17,755 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:17,755 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:17,755 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:17,755 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:17,755 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:17,755 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:17,763 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:17,763 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:17,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:17,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:17,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:17,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:17,763 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:17,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:17,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:17,764 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:17,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:17,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:17,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:17,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:17,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:17,767 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:17,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:17,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,769 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,769 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:17,769 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:17,769 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,769 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:17,769 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:17,769 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:17,769 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:17,769 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:17,769 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:17,769 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:17,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:17,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:17,769 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:17,769 DEBUG [Thread-831] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:17,770 DEBUG [Thread-831] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:17,770 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:17,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:17,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:17,870 DEBUG [Thread-831] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:17,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:17,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:17,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:17,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:17,873 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:17,873 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:17,873 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:17,873 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:17,874 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:17,874 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:17,875 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:17,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:17,875 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:17,875 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:17,875 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:17,876 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:17,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:17,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:17,876 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:17,876 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:17,876 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:17,876 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:17,876 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:17,876 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:17,876 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:17,876 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:17,877 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:17,877 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. started... 2023-04-12 15:54:17,877 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. as already flushing 2023-04-12 15:54:17,877 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:17,877 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:17,877 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:17,877 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,877 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:17,877 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:17,882 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,882 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:17,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:17,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:17,883 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:17,883 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:17,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:17,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:17,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:17,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:17,884 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:17,884 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e4d4882[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:17,884 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:17,884 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:17,886 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:17,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:17,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:17,886 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:17,886 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:17,886 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:17,886 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,886 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:17,887 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:17,887 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,887 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:17,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:17,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:17,887 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:17,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:17,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:17,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:17,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:17,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:17,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:17,889 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:17,889 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:17,889 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:17,889 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:17,889 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:17,889 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:17,891 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:17,891 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:17,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:17,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:17,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:17,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:17,891 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:17,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:17,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:17,891 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:17,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:17,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:17,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:17,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:17,892 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=216 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/C/9de195298436425e9801f8848b24ec6e 2023-04-12 15:54:17,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:17,893 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:17,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:17,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,898 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/A/3d54ed87c8084576afaf5a7e004de4ea as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/3d54ed87c8084576afaf5a7e004de4ea 2023-04-12 15:54:17,910 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/3d54ed87c8084576afaf5a7e004de4ea, entries=150, sequenceid=216, filesize=25.3 K 2023-04-12 15:54:17,911 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,911 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:17,911 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:17,911 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:17,911 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:17,911 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:17,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:17,911 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:17,911 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/B/9bf16651b56c467eb714602de7d72c09 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/9bf16651b56c467eb714602de7d72c09 2023-04-12 15:54:17,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:17,911 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:17,912 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:17,911 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:17,911 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:17,911 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:17,913 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:17,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:17,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:17,913 DEBUG [Thread-831] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:17,913 DEBUG [Thread-831] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:17,927 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/9bf16651b56c467eb714602de7d72c09, entries=150, sequenceid=216, filesize=11.8 K 2023-04-12 15:54:17,928 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/C/9de195298436425e9801f8848b24ec6e as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/9de195298436425e9801f8848b24ec6e 2023-04-12 15:54:17,938 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/9de195298436425e9801f8848b24ec6e, entries=150, sequenceid=216, filesize=11.8 K 2023-04-12 15:54:17,939 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for ca92eda5f752c4c8f5e2f01f66bc73d8 in 904ms, sequenceid=216, compaction requested=true 2023-04-12 15:54:17,939 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ca92eda5f752c4c8f5e2f01f66bc73d8: 2023-04-12 15:54:17,940 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 15:54:17,941 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:54:17,941 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:54:17,941 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:54:17,941 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 15:54:17,942 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 77996 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 15:54:17,942 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1912): ca92eda5f752c4c8f5e2f01f66bc73d8/A is initiating minor compaction (all files) 2023-04-12 15:54:17,942 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ca92eda5f752c4c8f5e2f01f66bc73d8/A in TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:17,942 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/7ff490ee87994b168395323b44b276ec, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/5c76f100bfc040a9b61197c36c380c46, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/3d54ed87c8084576afaf5a7e004de4ea] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp, totalSize=76.2 K 2023-04-12 15:54:17,943 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 7ff490ee87994b168395323b44b276ec, keycount=150, bloomtype=ROW, size=25.6 K, encoding=NONE, compression=NONE, seqNum=150, earliestPutTs=1681314854576 2023-04-12 15:54:17,952 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 5c76f100bfc040a9b61197c36c380c46, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1681314854779 2023-04-12 15:54:17,952 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36416 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 15:54:17,953 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1912): ca92eda5f752c4c8f5e2f01f66bc73d8/B is initiating minor compaction (all files) 2023-04-12 15:54:17,953 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ca92eda5f752c4c8f5e2f01f66bc73d8/B in TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:17,953 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/83437973030343d19c239e466a565a4c, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/d1413f20510a46dc9d1d84e13ecb0083, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/9bf16651b56c467eb714602de7d72c09] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp, totalSize=35.6 K 2023-04-12 15:54:17,953 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 3d54ed87c8084576afaf5a7e004de4ea, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=216, earliestPutTs=1681314855919 2023-04-12 15:54:17,953 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 83437973030343d19c239e466a565a4c, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=150, earliestPutTs=1681314854576 2023-04-12 15:54:17,954 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting d1413f20510a46dc9d1d84e13ecb0083, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1681314854779 2023-04-12 15:54:17,955 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 9bf16651b56c467eb714602de7d72c09, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=216, earliestPutTs=1681314855919 2023-04-12 15:54:17,973 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] throttle.PressureAwareThroughputController(145): ca92eda5f752c4c8f5e2f01f66bc73d8#B#compaction#139 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:54:17,980 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] throttle.PressureAwareThroughputController(145): ca92eda5f752c4c8f5e2f01f66bc73d8#A#compaction#138 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:54:18,013 DEBUG [Thread-831] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:18,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:18,015 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:18,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:18,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:18,016 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:18,016 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:18,016 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:18,016 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:18,017 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:18,017 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:18,018 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:18,018 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:18,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:18,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:18,018 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:18,018 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:18,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:18,019 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:18,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:18,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:18,019 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:18,019 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:18,019 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:18,020 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:18,020 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:18,020 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:18,020 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:18,020 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:18,020 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:18,020 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. started... 2023-04-12 15:54:18,020 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ca92eda5f752c4c8f5e2f01f66bc73d8 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-04-12 15:54:18,020 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca92eda5f752c4c8f5e2f01f66bc73d8, store=A 2023-04-12 15:54:18,020 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:54:18,020 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca92eda5f752c4c8f5e2f01f66bc73d8, store=B 2023-04-12 15:54:18,020 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:54:18,020 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca92eda5f752c4c8f5e2f01f66bc73d8, store=C 2023-04-12 15:54:18,021 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:54:18,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on ca92eda5f752c4c8f5e2f01f66bc73d8 2023-04-12 15:54:18,204 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. as already flushing 2023-04-12 15:54:18,211 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:18,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34066 deadline: 1681314918208, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:18,211 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:18,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34094 deadline: 1681314918211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:18,212 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:18,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34134 deadline: 1681314918211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:18,212 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:18,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34122 deadline: 1681314918212, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:18,313 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:18,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34066 deadline: 1681314918312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:18,314 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:18,315 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:18,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34094 deadline: 1681314918313, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:18,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34134 deadline: 1681314918313, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:18,394 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/B/22856aac047f40a280698be347f4fe95 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/22856aac047f40a280698be347f4fe95 2023-04-12 15:54:18,400 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ca92eda5f752c4c8f5e2f01f66bc73d8/B of ca92eda5f752c4c8f5e2f01f66bc73d8 into 22856aac047f40a280698be347f4fe95(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:54:18,400 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ca92eda5f752c4c8f5e2f01f66bc73d8: 2023-04-12 15:54:18,400 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8., storeName=ca92eda5f752c4c8f5e2f01f66bc73d8/B, priority=13, startTime=1681314857941; duration=0sec 2023-04-12 15:54:18,400 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:54:18,400 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 15:54:18,401 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36416 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 15:54:18,401 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1912): ca92eda5f752c4c8f5e2f01f66bc73d8/C is initiating minor compaction (all files) 2023-04-12 15:54:18,401 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ca92eda5f752c4c8f5e2f01f66bc73d8/C in TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:18,402 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/eaaeb56b20b14f5fb63fc8680ac76872, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/c6bef8e944eb405aa018c1fee08f16d6, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/9de195298436425e9801f8848b24ec6e] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp, totalSize=35.6 K 2023-04-12 15:54:18,402 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting eaaeb56b20b14f5fb63fc8680ac76872, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=150, earliestPutTs=1681314854576 2023-04-12 15:54:18,402 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting c6bef8e944eb405aa018c1fee08f16d6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1681314854779 2023-04-12 15:54:18,403 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 9de195298436425e9801f8848b24ec6e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=216, earliestPutTs=1681314855919 2023-04-12 15:54:18,425 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] throttle.PressureAwareThroughputController(145): ca92eda5f752c4c8f5e2f01f66bc73d8#C#compaction#141 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:54:18,440 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230412b5478d3adf45440dba4f53e6c1756db4 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412b5478d3adf45440dba4f53e6c1756db4 2023-04-12 15:54:18,456 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=253, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/A/c6f88585101f45b4a232ba5b2f193cbe 2023-04-12 15:54:18,457 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/C/cdb9d721dd7246fe98d1f16a35f98b4f as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/cdb9d721dd7246fe98d1f16a35f98b4f 2023-04-12 15:54:18,463 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ca92eda5f752c4c8f5e2f01f66bc73d8/C of ca92eda5f752c4c8f5e2f01f66bc73d8 into cdb9d721dd7246fe98d1f16a35f98b4f(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:54:18,463 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ca92eda5f752c4c8f5e2f01f66bc73d8: 2023-04-12 15:54:18,463 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8., storeName=ca92eda5f752c4c8f5e2f01f66bc73d8/C, priority=13, startTime=1681314857941; duration=0sec 2023-04-12 15:54:18,464 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:54:18,519 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:18,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34066 deadline: 1681314918516, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:18,519 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:18,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34094 deadline: 1681314918517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:18,519 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:18,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34134 deadline: 1681314918517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:18,821 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:18,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34094 deadline: 1681314918820, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:18,821 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:18,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34066 deadline: 1681314918820, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:18,822 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:18,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34134 deadline: 1681314918821, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:18,868 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=253 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/B/f5960bdffe644342a31f54fbbb56e08c 2023-04-12 15:54:19,201 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/A/70f4dcfa2997461b8a9027a10bbc2a90 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/70f4dcfa2997461b8a9027a10bbc2a90 2023-04-12 15:54:19,206 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ca92eda5f752c4c8f5e2f01f66bc73d8/A of ca92eda5f752c4c8f5e2f01f66bc73d8 into 70f4dcfa2997461b8a9027a10bbc2a90(size=25.7 K), total size for store is 25.7 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-12 15:54:19,206 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ca92eda5f752c4c8f5e2f01f66bc73d8: 2023-04-12 15:54:19,206 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8., storeName=ca92eda5f752c4c8f5e2f01f66bc73d8/A, priority=13, startTime=1681314857940; duration=1sec 2023-04-12 15:54:19,206 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:54:19,225 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:19,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34122 deadline: 1681314919222, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:19,280 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=253 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/C/ed513faf841744dd895bbf0cbbfec864 2023-04-12 15:54:19,285 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/A/c6f88585101f45b4a232ba5b2f193cbe as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/c6f88585101f45b4a232ba5b2f193cbe 2023-04-12 15:54:19,289 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/c6f88585101f45b4a232ba5b2f193cbe, entries=150, sequenceid=253, filesize=25.3 K 2023-04-12 15:54:19,290 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/B/f5960bdffe644342a31f54fbbb56e08c as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/f5960bdffe644342a31f54fbbb56e08c 2023-04-12 15:54:19,295 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/f5960bdffe644342a31f54fbbb56e08c, entries=150, sequenceid=253, filesize=11.8 K 2023-04-12 15:54:19,295 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/C/ed513faf841744dd895bbf0cbbfec864 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/ed513faf841744dd895bbf0cbbfec864 2023-04-12 15:54:19,300 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/ed513faf841744dd895bbf0cbbfec864, entries=150, sequenceid=253, filesize=11.8 K 2023-04-12 15:54:19,301 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=40.25 KB/41220 for ca92eda5f752c4c8f5e2f01f66bc73d8 in 1281ms, sequenceid=253, compaction requested=false 2023-04-12 15:54:19,301 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ca92eda5f752c4c8f5e2f01f66bc73d8: 2023-04-12 15:54:19,301 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:19,301 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:19,301 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:19,302 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:19,302 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:19,302 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:19,303 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:19,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:19,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:19,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:19,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:19,304 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:19,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:19,304 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:19,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:19,304 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:19,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:19,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:19,305 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:19,305 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36f1f970[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:19,305 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:19,305 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:19,306 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:19,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:19,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:19,306 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:19,306 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:19,306 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:19,307 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:19,307 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:19,308 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:19,308 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:19,308 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:19,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:19,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:19,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:19,308 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:19,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:19,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:19,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:19,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:19,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:19,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:19,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:19,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:19,311 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:19,311 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:19,311 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:19,311 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:19,311 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:19,311 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:19,313 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:19,313 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:19,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:19,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:19,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:19,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:19,313 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:19,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:19,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:19,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:19,314 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:19,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:19,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:19,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:19,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:19,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:19,315 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:19,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:19,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:19,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:19,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:19,324 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:19,324 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:19,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:19,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:19,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:19,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:19,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on ca92eda5f752c4c8f5e2f01f66bc73d8 2023-04-12 15:54:19,327 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ca92eda5f752c4c8f5e2f01f66bc73d8 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-12 15:54:19,327 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca92eda5f752c4c8f5e2f01f66bc73d8, store=A 2023-04-12 15:54:19,328 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:19,328 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca92eda5f752c4c8f5e2f01f66bc73d8, store=B 2023-04-12 15:54:19,328 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:19,328 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca92eda5f752c4c8f5e2f01f66bc73d8, store=C 2023-04-12 15:54:19,328 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:19,328 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:19,328 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:19,328 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:19,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:19,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:19,328 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:19,328 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:19,328 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:19,328 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:19,328 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:19,328 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:19,328 DEBUG [Thread-831] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:19,328 DEBUG [Thread-831] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:19,350 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304127abb732ee36f4b4ba914030bed87d15f to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304127abb732ee36f4b4ba914030bed87d15f 2023-04-12 15:54:19,355 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:19,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34134 deadline: 1681314919351, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:19,356 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:19,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34094 deadline: 1681314919352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:19,356 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:19,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34066 deadline: 1681314919356, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:19,361 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=268, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/A/036b454ea2034330a02f23811342347b 2023-04-12 15:54:19,378 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=268 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/B/9e71c39221624c21aaf45b199245c1e7 2023-04-12 15:54:19,406 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=268 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/C/cb299058d1de45b7b5158c8881775503 2023-04-12 15:54:19,412 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/A/036b454ea2034330a02f23811342347b as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/036b454ea2034330a02f23811342347b 2023-04-12 15:54:19,417 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/036b454ea2034330a02f23811342347b, entries=200, sequenceid=268, filesize=32.3 K 2023-04-12 15:54:19,418 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/B/9e71c39221624c21aaf45b199245c1e7 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/9e71c39221624c21aaf45b199245c1e7 2023-04-12 15:54:19,424 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/9e71c39221624c21aaf45b199245c1e7, entries=150, sequenceid=268, filesize=11.9 K 2023-04-12 15:54:19,425 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/C/cb299058d1de45b7b5158c8881775503 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/cb299058d1de45b7b5158c8881775503 2023-04-12 15:54:19,429 DEBUG [Thread-831] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:19,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:19,431 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:19,432 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/cb299058d1de45b7b5158c8881775503, entries=150, sequenceid=268, filesize=11.9 K 2023-04-12 15:54:19,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:19,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:19,433 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:19,433 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:19,433 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:19,433 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:19,433 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for ca92eda5f752c4c8f5e2f01f66bc73d8 in 106ms, sequenceid=268, compaction requested=true 2023-04-12 15:54:19,433 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ca92eda5f752c4c8f5e2f01f66bc73d8: 2023-04-12 15:54:19,433 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:54:19,433 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-12 15:54:19,433 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-04-12 15:54:19,434 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 15:54:19,434 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 15:54:19,437 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85288 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 15:54:19,437 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1912): ca92eda5f752c4c8f5e2f01f66bc73d8/A is initiating minor compaction (all files) 2023-04-12 15:54:19,437 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ca92eda5f752c4c8f5e2f01f66bc73d8/A in TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:19,437 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/70f4dcfa2997461b8a9027a10bbc2a90, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/c6f88585101f45b4a232ba5b2f193cbe, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/036b454ea2034330a02f23811342347b] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp, totalSize=83.3 K 2023-04-12 15:54:19,437 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36668 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 15:54:19,437 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1912): ca92eda5f752c4c8f5e2f01f66bc73d8/C is initiating minor compaction (all files) 2023-04-12 15:54:19,437 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ca92eda5f752c4c8f5e2f01f66bc73d8/C in TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:19,437 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/cdb9d721dd7246fe98d1f16a35f98b4f, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/ed513faf841744dd895bbf0cbbfec864, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/cb299058d1de45b7b5158c8881775503] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp, totalSize=35.8 K 2023-04-12 15:54:19,439 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:19,439 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:19,439 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 70f4dcfa2997461b8a9027a10bbc2a90, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=216, earliestPutTs=1681314855919 2023-04-12 15:54:19,440 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting cdb9d721dd7246fe98d1f16a35f98b4f, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=216, earliestPutTs=1681314855919 2023-04-12 15:54:19,440 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:19,440 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:19,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:19,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:19,441 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting ed513faf841744dd895bbf0cbbfec864, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=253, earliestPutTs=1681314857073 2023-04-12 15:54:19,441 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting c6f88585101f45b4a232ba5b2f193cbe, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=253, earliestPutTs=1681314857073 2023-04-12 15:54:19,441 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:19,441 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:19,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:19,441 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:19,442 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting cb299058d1de45b7b5158c8881775503, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=268, earliestPutTs=1681314858209 2023-04-12 15:54:19,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:19,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:19,442 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:19,442 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 036b454ea2034330a02f23811342347b, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=268, earliestPutTs=1681314858209 2023-04-12 15:54:19,442 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:19,442 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:19,443 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:19,443 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:19,443 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:19,444 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:19,444 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:19,444 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:19,444 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. started... 2023-04-12 15:54:19,444 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ca92eda5f752c4c8f5e2f01f66bc73d8 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-12 15:54:19,444 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca92eda5f752c4c8f5e2f01f66bc73d8, store=A 2023-04-12 15:54:19,444 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:19,445 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca92eda5f752c4c8f5e2f01f66bc73d8, store=B 2023-04-12 15:54:19,445 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:19,445 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca92eda5f752c4c8f5e2f01f66bc73d8, store=C 2023-04-12 15:54:19,445 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:19,456 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] throttle.PressureAwareThroughputController(145): ca92eda5f752c4c8f5e2f01f66bc73d8#C#compaction#147 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:54:19,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on ca92eda5f752c4c8f5e2f01f66bc73d8 2023-04-12 15:54:19,461 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. as already flushing 2023-04-12 15:54:19,472 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] throttle.PressureAwareThroughputController(145): ca92eda5f752c4c8f5e2f01f66bc73d8#A#compaction#149 average throughput is 1.41 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:54:19,478 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:19,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34066 deadline: 1681314919475, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:19,484 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:19,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34094 deadline: 1681314919476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:19,484 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304129963f0c94f084974b9f55178f3a1cc88 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304129963f0c94f084974b9f55178f3a1cc88 2023-04-12 15:54:19,484 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:19,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34134 deadline: 1681314919476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:19,504 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=288, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/A/abe818eaef4a4430b42c70a78f5d3cab 2023-04-12 15:54:19,520 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=288 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/B/380fa9ee9be9437a9299d780b4cc88b8 2023-04-12 15:54:19,528 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/A/1fe66f000dd448489d9c2e0b2b61f0c2 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/1fe66f000dd448489d9c2e0b2b61f0c2 2023-04-12 15:54:19,531 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=288 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/C/a31d3f51858349f09710fcf38bd09889 2023-04-12 15:54:19,534 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ca92eda5f752c4c8f5e2f01f66bc73d8/A of ca92eda5f752c4c8f5e2f01f66bc73d8 into 1fe66f000dd448489d9c2e0b2b61f0c2(size=25.9 K), total size for store is 25.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:54:19,534 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ca92eda5f752c4c8f5e2f01f66bc73d8: 2023-04-12 15:54:19,534 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8., storeName=ca92eda5f752c4c8f5e2f01f66bc73d8/A, priority=13, startTime=1681314859433; duration=0sec 2023-04-12 15:54:19,534 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:54:19,534 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 15:54:19,538 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/A/abe818eaef4a4430b42c70a78f5d3cab as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/abe818eaef4a4430b42c70a78f5d3cab 2023-04-12 15:54:19,538 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36668 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 15:54:19,538 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1912): ca92eda5f752c4c8f5e2f01f66bc73d8/B is initiating minor compaction (all files) 2023-04-12 15:54:19,538 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ca92eda5f752c4c8f5e2f01f66bc73d8/B in TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:19,538 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/22856aac047f40a280698be347f4fe95, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/f5960bdffe644342a31f54fbbb56e08c, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/9e71c39221624c21aaf45b199245c1e7] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp, totalSize=35.8 K 2023-04-12 15:54:19,539 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 22856aac047f40a280698be347f4fe95, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=216, earliestPutTs=1681314855919 2023-04-12 15:54:19,539 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting f5960bdffe644342a31f54fbbb56e08c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=253, earliestPutTs=1681314857073 2023-04-12 15:54:19,540 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 9e71c39221624c21aaf45b199245c1e7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=268, earliestPutTs=1681314858209 2023-04-12 15:54:19,543 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/abe818eaef4a4430b42c70a78f5d3cab, entries=150, sequenceid=288, filesize=25.4 K 2023-04-12 15:54:19,545 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/B/380fa9ee9be9437a9299d780b4cc88b8 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/380fa9ee9be9437a9299d780b4cc88b8 2023-04-12 15:54:19,549 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/380fa9ee9be9437a9299d780b4cc88b8, entries=150, sequenceid=288, filesize=11.9 K 2023-04-12 15:54:19,550 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] throttle.PressureAwareThroughputController(145): ca92eda5f752c4c8f5e2f01f66bc73d8#B#compaction#152 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:54:19,551 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/C/a31d3f51858349f09710fcf38bd09889 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/a31d3f51858349f09710fcf38bd09889 2023-04-12 15:54:19,560 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/a31d3f51858349f09710fcf38bd09889, entries=150, sequenceid=288, filesize=11.9 K 2023-04-12 15:54:19,561 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=67.09 KB/68700 for ca92eda5f752c4c8f5e2f01f66bc73d8 in 117ms, sequenceid=288, compaction requested=false 2023-04-12 15:54:19,561 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ca92eda5f752c4c8f5e2f01f66bc73d8: 2023-04-12 15:54:19,561 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:19,561 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:19,561 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:19,561 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:19,561 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:19,561 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:19,562 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:19,562 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:19,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:19,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:19,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:19,563 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:19,563 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:19,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:19,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:19,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:19,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:19,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:19,565 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:19,565 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7eccaf48[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:19,565 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:19,565 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:19,567 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:19,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:19,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:19,567 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:19,568 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:19,568 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:19,568 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:19,568 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:19,569 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:19,569 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:19,569 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:19,569 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:19,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:19,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:19,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:19,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:19,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:19,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:19,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:19,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:19,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:19,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:19,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:19,574 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:19,574 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:19,574 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:19,574 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:19,574 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:19,574 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:19,575 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:19,575 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:19,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:19,575 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:19,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:19,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:19,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:19,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:19,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:19,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:19,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:19,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:19,576 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:19,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:19,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:19,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:19,578 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:19,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:19,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:19,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:19,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:19,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on ca92eda5f752c4c8f5e2f01f66bc73d8 2023-04-12 15:54:19,581 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ca92eda5f752c4c8f5e2f01f66bc73d8 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-12 15:54:19,581 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca92eda5f752c4c8f5e2f01f66bc73d8, store=A 2023-04-12 15:54:19,581 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:19,581 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca92eda5f752c4c8f5e2f01f66bc73d8, store=B 2023-04-12 15:54:19,581 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:19,581 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca92eda5f752c4c8f5e2f01f66bc73d8, store=C 2023-04-12 15:54:19,581 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:19,581 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:19,581 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:19,581 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:19,581 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:19,581 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:19,581 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:19,581 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:19,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:19,581 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:19,581 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:19,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:19,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:19,582 DEBUG [Thread-831] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:19,582 DEBUG [Thread-831] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:19,582 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:19,582 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:19,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:19,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:19,607 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:19,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34066 deadline: 1681314919603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:19,607 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:19,608 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:19,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34134 deadline: 1681314919603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:19,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34094 deadline: 1681314919604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:19,682 DEBUG [Thread-831] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:19,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:19,684 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:19,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:19,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:19,685 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:19,685 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:19,685 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:19,685 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:19,685 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:19,686 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:19,688 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:19,688 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:19,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:19,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:19,688 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:19,688 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:19,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:19,689 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:19,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:19,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:19,689 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:19,690 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:19,690 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:19,690 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:19,690 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:19,690 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:19,690 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:19,690 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:19,690 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:19,690 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. started... 2023-04-12 15:54:19,690 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. as already flushing 2023-04-12 15:54:19,690 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:19,690 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:19,690 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:19,690 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:19,690 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:19,690 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:19,692 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:19,692 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:19,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:19,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:19,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:19,692 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:19,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:19,692 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:19,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:19,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:19,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:19,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:19,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:19,694 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:19,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5941dba2[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:19,694 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:19,702 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:19,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:19,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:19,702 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:19,702 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:19,702 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:19,702 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:19,702 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:19,704 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:19,704 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:19,704 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:19,704 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:19,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:19,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:19,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:19,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:19,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:19,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:19,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:19,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:19,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:19,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:19,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:19,707 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:19,707 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:19,707 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:19,707 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:19,707 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:19,707 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:19,709 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:19,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34066 deadline: 1681314919708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:19,709 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:19,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34094 deadline: 1681314919709, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:19,709 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:19,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34134 deadline: 1681314919709, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:19,718 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:19,718 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:19,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:19,718 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:19,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:19,719 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:19,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:19,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:19,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:19,720 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:19,722 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:19,722 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:19,722 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:19,722 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:19,722 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:19,722 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:19,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:19,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:19,722 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:19,722 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:19,722 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:19,722 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:19,722 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:19,722 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:19,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:19,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:19,723 DEBUG [Thread-831] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:19,723 DEBUG [Thread-831] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:19,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:19,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:19,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:19,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:19,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:19,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:19,725 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:19,823 DEBUG [Thread-831] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:19,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:19,826 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:19,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:19,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:19,827 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:19,827 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:19,827 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:19,827 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:19,828 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:19,828 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:19,829 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:19,829 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:19,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:19,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:19,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:19,830 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:19,830 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:19,830 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:19,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:19,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:19,831 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:19,831 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:19,831 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:19,831 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:19,831 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:19,831 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:19,831 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:19,831 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:19,831 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:19,831 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. started... 2023-04-12 15:54:19,831 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. as already flushing 2023-04-12 15:54:19,831 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:19,832 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:19,832 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:19,832 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:19,832 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:19,832 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:19,834 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:19,834 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:19,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:19,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:19,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:19,834 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:19,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:19,834 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:19,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:19,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:19,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:19,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:19,836 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:19,836 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5dcc8150[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:19,836 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:19,836 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:19,837 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:19,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:19,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:19,837 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:19,837 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:19,837 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:19,838 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:19,838 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:19,840 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:19,840 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:19,840 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:19,840 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:19,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:19,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:19,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:19,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:19,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:19,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:19,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:19,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:19,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:19,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:19,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:19,842 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:19,842 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:19,842 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:19,842 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:19,842 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:19,842 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:19,844 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:19,844 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:19,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:19,844 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:19,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:19,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:19,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:19,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:19,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:19,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:19,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:19,845 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:19,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:19,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:19,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:19,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:19,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:19,846 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:19,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:19,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:19,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:19,850 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:19,850 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:19,850 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:19,850 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:19,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:19,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:19,850 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:19,850 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:19,850 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:19,851 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:19,851 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:19,851 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:19,851 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:19,850 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:19,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:19,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:19,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:19,851 DEBUG [Thread-831] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:19,851 DEBUG [Thread-831] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:19,891 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/C/128e5a99e00b4744a8632b8a68c5d339 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/128e5a99e00b4744a8632b8a68c5d339 2023-04-12 15:54:19,897 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ca92eda5f752c4c8f5e2f01f66bc73d8/C of ca92eda5f752c4c8f5e2f01f66bc73d8 into 128e5a99e00b4744a8632b8a68c5d339(size=12.4 K), total size for store is 24.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:54:19,897 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ca92eda5f752c4c8f5e2f01f66bc73d8: 2023-04-12 15:54:19,897 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8., storeName=ca92eda5f752c4c8f5e2f01f66bc73d8/C, priority=13, startTime=1681314859433; duration=0sec 2023-04-12 15:54:19,897 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:54:19,912 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:19,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34094 deadline: 1681314919910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:19,912 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:19,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34066 deadline: 1681314919911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:19,913 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:19,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34134 deadline: 1681314919911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:19,951 DEBUG [Thread-831] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:19,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:19,953 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:19,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:19,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:19,955 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:19,955 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:19,955 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:19,955 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:19,955 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:19,955 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:19,956 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:19,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:19,956 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:19,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:19,957 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:19,957 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:19,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:19,957 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:19,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:19,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:19,958 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:19,958 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:19,958 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:19,958 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:19,958 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:19,958 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:19,958 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:19,958 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:19,958 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:19,958 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. started... 2023-04-12 15:54:19,958 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. as already flushing 2023-04-12 15:54:19,958 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:19,958 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:19,958 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:19,958 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:19,958 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:19,958 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:19,960 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:19,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:19,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:19,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:19,960 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:19,960 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:19,960 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:19,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:19,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:19,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:19,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:19,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:19,962 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:19,962 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64cfd9e0[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:19,962 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:19,962 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:19,963 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:19,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:19,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:19,963 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:19,964 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:19,964 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:19,964 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:19,964 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:19,965 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/B/3f83afdc81fe4a78b07feb0d46df5e59 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/3f83afdc81fe4a78b07feb0d46df5e59 2023-04-12 15:54:19,967 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:19,967 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:19,967 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:19,968 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:19,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:19,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:19,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:19,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:19,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:19,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:19,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:19,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:19,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:19,972 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ca92eda5f752c4c8f5e2f01f66bc73d8/B of ca92eda5f752c4c8f5e2f01f66bc73d8 into 3f83afdc81fe4a78b07feb0d46df5e59(size=12.4 K), total size for store is 24.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:54:19,972 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ca92eda5f752c4c8f5e2f01f66bc73d8: 2023-04-12 15:54:19,972 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8., storeName=ca92eda5f752c4c8f5e2f01f66bc73d8/B, priority=13, startTime=1681314859433; duration=0sec 2023-04-12 15:54:19,972 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:54:19,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:19,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:19,973 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:19,973 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:19,973 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:19,973 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:19,973 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:19,973 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:19,974 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:19,974 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:19,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:19,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:19,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:19,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:19,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:19,974 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:19,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:19,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:19,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:19,975 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:19,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:19,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:19,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:19,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:19,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:19,977 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:19,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:19,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:19,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:19,980 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:19,980 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:19,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:19,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:19,980 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:19,980 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:19,980 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:19,980 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:19,980 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:19,980 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:19,980 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:19,980 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:19,980 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:19,980 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:19,981 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:19,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:19,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:19,981 DEBUG [Thread-831] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:19,981 DEBUG [Thread-831] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:19,997 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230412d018f45a34324780bc590cd43e64f3a4 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412d018f45a34324780bc590cd43e64f3a4 2023-04-12 15:54:20,011 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=303, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/A/3d520b66e0a7428bae5f878b06b4f65d 2023-04-12 15:54:20,030 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=303 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/B/6da84a2049ff425a82b8a58b489dba73 2023-04-12 15:54:20,050 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=303 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/C/37acf76c211043079819a4bb57ae55e7 2023-04-12 15:54:20,056 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/A/3d520b66e0a7428bae5f878b06b4f65d as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/3d520b66e0a7428bae5f878b06b4f65d 2023-04-12 15:54:20,062 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/3d520b66e0a7428bae5f878b06b4f65d, entries=200, sequenceid=303, filesize=32.3 K 2023-04-12 15:54:20,063 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/B/6da84a2049ff425a82b8a58b489dba73 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/6da84a2049ff425a82b8a58b489dba73 2023-04-12 15:54:20,068 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/6da84a2049ff425a82b8a58b489dba73, entries=150, sequenceid=303, filesize=11.9 K 2023-04-12 15:54:20,069 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/C/37acf76c211043079819a4bb57ae55e7 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/37acf76c211043079819a4bb57ae55e7 2023-04-12 15:54:20,075 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/37acf76c211043079819a4bb57ae55e7, entries=150, sequenceid=303, filesize=11.9 K 2023-04-12 15:54:20,077 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for ca92eda5f752c4c8f5e2f01f66bc73d8 in 496ms, sequenceid=303, compaction requested=true 2023-04-12 15:54:20,077 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ca92eda5f752c4c8f5e2f01f66bc73d8: 2023-04-12 15:54:20,077 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:54:20,077 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 15:54:20,077 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 15:54:20,077 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:54:20,077 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:54:20,078 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85690 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 15:54:20,079 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37070 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 15:54:20,079 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1912): ca92eda5f752c4c8f5e2f01f66bc73d8/A is initiating minor compaction (all files) 2023-04-12 15:54:20,079 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1912): ca92eda5f752c4c8f5e2f01f66bc73d8/B is initiating minor compaction (all files) 2023-04-12 15:54:20,079 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ca92eda5f752c4c8f5e2f01f66bc73d8/A in TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:20,079 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ca92eda5f752c4c8f5e2f01f66bc73d8/B in TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:20,079 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/1fe66f000dd448489d9c2e0b2b61f0c2, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/abe818eaef4a4430b42c70a78f5d3cab, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/3d520b66e0a7428bae5f878b06b4f65d] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp, totalSize=83.7 K 2023-04-12 15:54:20,079 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/3f83afdc81fe4a78b07feb0d46df5e59, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/380fa9ee9be9437a9299d780b4cc88b8, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/6da84a2049ff425a82b8a58b489dba73] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp, totalSize=36.2 K 2023-04-12 15:54:20,080 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 1fe66f000dd448489d9c2e0b2b61f0c2, keycount=150, bloomtype=ROW, size=25.9 K, encoding=NONE, compression=NONE, seqNum=268, earliestPutTs=1681314858209 2023-04-12 15:54:20,080 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting abe818eaef4a4430b42c70a78f5d3cab, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=288, earliestPutTs=1681314859336 2023-04-12 15:54:20,080 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 3f83afdc81fe4a78b07feb0d46df5e59, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=268, earliestPutTs=1681314858209 2023-04-12 15:54:20,081 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 3d520b66e0a7428bae5f878b06b4f65d, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=303, earliestPutTs=1681314859474 2023-04-12 15:54:20,081 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 380fa9ee9be9437a9299d780b4cc88b8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=288, earliestPutTs=1681314859336 2023-04-12 15:54:20,081 DEBUG [Thread-831] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:20,081 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 6da84a2049ff425a82b8a58b489dba73, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=303, earliestPutTs=1681314859474 2023-04-12 15:54:20,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:20,084 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:20,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:20,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:20,088 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:20,088 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:20,088 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:20,088 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:20,089 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:20,089 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:20,090 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:20,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:20,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:20,091 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:20,092 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:20,092 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:20,092 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] throttle.PressureAwareThroughputController(145): ca92eda5f752c4c8f5e2f01f66bc73d8#B#compaction#156 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:54:20,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:20,096 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:20,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:20,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:20,097 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:20,105 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:20,105 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:20,105 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:20,105 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:20,105 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:20,105 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:20,105 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:20,105 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:20,105 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. started... 2023-04-12 15:54:20,106 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ca92eda5f752c4c8f5e2f01f66bc73d8 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-12 15:54:20,106 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca92eda5f752c4c8f5e2f01f66bc73d8, store=A 2023-04-12 15:54:20,106 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:20,106 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca92eda5f752c4c8f5e2f01f66bc73d8, store=B 2023-04-12 15:54:20,106 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:20,106 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca92eda5f752c4c8f5e2f01f66bc73d8, store=C 2023-04-12 15:54:20,106 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] throttle.PressureAwareThroughputController(145): ca92eda5f752c4c8f5e2f01f66bc73d8#A#compaction#157 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:54:20,106 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:20,139 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/B/8b50d0c3dcdb423d92caa6e6db2b61c9 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/8b50d0c3dcdb423d92caa6e6db2b61c9 2023-04-12 15:54:20,158 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ca92eda5f752c4c8f5e2f01f66bc73d8/B of ca92eda5f752c4c8f5e2f01f66bc73d8 into 8b50d0c3dcdb423d92caa6e6db2b61c9(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:54:20,158 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ca92eda5f752c4c8f5e2f01f66bc73d8: 2023-04-12 15:54:20,158 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8., storeName=ca92eda5f752c4c8f5e2f01f66bc73d8/B, priority=13, startTime=1681314860077; duration=0sec 2023-04-12 15:54:20,158 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:54:20,158 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 15:54:20,160 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37070 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 15:54:20,161 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1912): ca92eda5f752c4c8f5e2f01f66bc73d8/C is initiating minor compaction (all files) 2023-04-12 15:54:20,161 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ca92eda5f752c4c8f5e2f01f66bc73d8/C in TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:20,161 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/128e5a99e00b4744a8632b8a68c5d339, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/a31d3f51858349f09710fcf38bd09889, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/37acf76c211043079819a4bb57ae55e7] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp, totalSize=36.2 K 2023-04-12 15:54:20,161 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 128e5a99e00b4744a8632b8a68c5d339, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=268, earliestPutTs=1681314858209 2023-04-12 15:54:20,162 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting a31d3f51858349f09710fcf38bd09889, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=288, earliestPutTs=1681314859336 2023-04-12 15:54:20,162 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 37acf76c211043079819a4bb57ae55e7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=303, earliestPutTs=1681314859474 2023-04-12 15:54:20,188 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] throttle.PressureAwareThroughputController(145): ca92eda5f752c4c8f5e2f01f66bc73d8#C#compaction#159 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:54:20,198 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/C/c2f49a8f5d6d44558e087af982d82ad8 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/c2f49a8f5d6d44558e087af982d82ad8 2023-04-12 15:54:20,203 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ca92eda5f752c4c8f5e2f01f66bc73d8/C of ca92eda5f752c4c8f5e2f01f66bc73d8 into c2f49a8f5d6d44558e087af982d82ad8(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:54:20,203 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ca92eda5f752c4c8f5e2f01f66bc73d8: 2023-04-12 15:54:20,203 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8., storeName=ca92eda5f752c4c8f5e2f01f66bc73d8/C, priority=13, startTime=1681314860077; duration=0sec 2023-04-12 15:54:20,203 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:54:20,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on ca92eda5f752c4c8f5e2f01f66bc73d8 2023-04-12 15:54:20,216 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. as already flushing 2023-04-12 15:54:20,228 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:20,228 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:20,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34134 deadline: 1681314920224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:20,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34094 deadline: 1681314920224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:20,228 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:20,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34066 deadline: 1681314920228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:20,332 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:20,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34134 deadline: 1681314920329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:20,332 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:20,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34094 deadline: 1681314920329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:20,332 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:20,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34066 deadline: 1681314920329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:20,534 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:20,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34134 deadline: 1681314920533, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:20,534 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:20,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34094 deadline: 1681314920533, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:20,535 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:20,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34066 deadline: 1681314920534, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:20,587 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230412bda3e6d797af4e30b2f11f91932e1596 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412bda3e6d797af4e30b2f11f91932e1596 2023-04-12 15:54:20,837 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:20,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34094 deadline: 1681314920835, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:20,837 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:20,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34066 deadline: 1681314920836, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:20,837 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:20,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34134 deadline: 1681314920837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:20,997 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=323, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/A/720dfb88249b4815bd19e329e3c100cb 2023-04-12 15:54:20,999 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/A/ca2d4af540b24fb38a0fa858ddbd6042 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/ca2d4af540b24fb38a0fa858ddbd6042 2023-04-12 15:54:21,007 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ca92eda5f752c4c8f5e2f01f66bc73d8/A of ca92eda5f752c4c8f5e2f01f66bc73d8 into ca2d4af540b24fb38a0fa858ddbd6042(size=26.0 K), total size for store is 26.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:54:21,007 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ca92eda5f752c4c8f5e2f01f66bc73d8: 2023-04-12 15:54:21,007 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8., storeName=ca92eda5f752c4c8f5e2f01f66bc73d8/A, priority=13, startTime=1681314860077; duration=0sec 2023-04-12 15:54:21,007 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:54:21,233 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:21,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34122 deadline: 1681314921231, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:21,235 DEBUG [Thread-821] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4144 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8., hostname=jenkins-hbase4.apache.org,38905,1681314818739, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-12 15:54:21,339 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:21,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34094 deadline: 1681314921338, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:21,340 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:21,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34134 deadline: 1681314921339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:21,342 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:21,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34066 deadline: 1681314921340, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:21,415 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=323 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/B/5ac59dfdb9d44efd8f790ec7d0631ca1 2023-04-12 15:54:21,826 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=323 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/C/49ff5b64306148729f8bdbe9ddc846f9 2023-04-12 15:54:21,831 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/A/720dfb88249b4815bd19e329e3c100cb as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/720dfb88249b4815bd19e329e3c100cb 2023-04-12 15:54:21,835 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/720dfb88249b4815bd19e329e3c100cb, entries=150, sequenceid=323, filesize=25.4 K 2023-04-12 15:54:21,836 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/B/5ac59dfdb9d44efd8f790ec7d0631ca1 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/5ac59dfdb9d44efd8f790ec7d0631ca1 2023-04-12 15:54:21,839 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/5ac59dfdb9d44efd8f790ec7d0631ca1, entries=150, sequenceid=323, filesize=11.9 K 2023-04-12 15:54:21,840 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/C/49ff5b64306148729f8bdbe9ddc846f9 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/49ff5b64306148729f8bdbe9ddc846f9 2023-04-12 15:54:21,844 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/49ff5b64306148729f8bdbe9ddc846f9, entries=150, sequenceid=323, filesize=11.9 K 2023-04-12 15:54:21,845 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for ca92eda5f752c4c8f5e2f01f66bc73d8 in 1739ms, sequenceid=323, compaction requested=false 2023-04-12 15:54:21,845 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ca92eda5f752c4c8f5e2f01f66bc73d8: 2023-04-12 15:54:21,845 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:21,845 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:21,845 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:21,845 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:21,845 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:21,845 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:21,847 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:21,847 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:21,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:21,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:21,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:21,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:21,848 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:21,848 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:21,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:21,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:21,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:21,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:21,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:21,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f629cde[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:21,849 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:21,849 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:21,850 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:21,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:21,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:21,851 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:21,851 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:21,851 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:21,851 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:21,851 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:21,852 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:21,852 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:21,852 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:21,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:21,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:21,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:21,852 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:21,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:21,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:21,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:21,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:21,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:21,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:21,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:21,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:21,854 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:21,854 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:21,855 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:21,855 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:21,855 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:21,855 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:21,859 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:21,859 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:21,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:21,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:21,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:21,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:21,859 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:21,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:21,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:21,860 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:21,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:21,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:21,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:21,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:21,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:21,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:21,861 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:21,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:21,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:21,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:21,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:21,863 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:21,863 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:21,863 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:21,863 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:21,863 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:21,863 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:21,863 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:21,863 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:21,863 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:21,863 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:21,863 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:21,863 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:21,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:21,863 DEBUG [Thread-831] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:21,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:21,863 DEBUG [Thread-831] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:21,863 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:21,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:21,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:21,963 DEBUG [Thread-831] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:21,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:21,965 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:21,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:21,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:21,967 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:21,967 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:21,967 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:21,967 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:21,967 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:21,967 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:21,969 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:21,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:21,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:21,969 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:21,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:21,969 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:21,969 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:21,970 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:21,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:21,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:21,970 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:21,970 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:21,970 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:21,970 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:21,970 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:21,970 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:21,970 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:21,970 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:21,971 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:21,971 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. started... 2023-04-12 15:54:21,971 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ca92eda5f752c4c8f5e2f01f66bc73d8 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-12 15:54:21,971 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca92eda5f752c4c8f5e2f01f66bc73d8, store=A 2023-04-12 15:54:21,971 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:21,971 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca92eda5f752c4c8f5e2f01f66bc73d8, store=B 2023-04-12 15:54:21,971 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:21,971 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca92eda5f752c4c8f5e2f01f66bc73d8, store=C 2023-04-12 15:54:21,971 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:22,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on ca92eda5f752c4c8f5e2f01f66bc73d8 2023-04-12 15:54:22,343 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. as already flushing 2023-04-12 15:54:22,360 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:22,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34134 deadline: 1681314922357, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:22,361 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:22,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34094 deadline: 1681314922358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:22,363 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:22,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34066 deadline: 1681314922361, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:22,386 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230412414669363df940c587bfc1d662b24e5e to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412414669363df940c587bfc1d662b24e5e 2023-04-12 15:54:22,392 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=340, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/A/d5c882c855c64cbb8188d8a4d4da2ef9 2023-04-12 15:54:22,462 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:22,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34094 deadline: 1681314922461, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:22,462 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:22,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34134 deadline: 1681314922461, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:22,465 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:22,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34066 deadline: 1681314922464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:22,664 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:22,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34094 deadline: 1681314922664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:22,666 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:22,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34134 deadline: 1681314922664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:22,669 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:22,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34066 deadline: 1681314922667, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:22,813 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=340 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/B/0de51cfd163947f983b88582735cdeb1 2023-04-12 15:54:22,967 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:22,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34094 deadline: 1681314922965, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:22,970 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:22,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34134 deadline: 1681314922969, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:22,970 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:22,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34066 deadline: 1681314922970, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:23,225 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=340 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/C/2e626e11a9c44828b4bdac944edba0d4 2023-04-12 15:54:23,230 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/A/d5c882c855c64cbb8188d8a4d4da2ef9 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/d5c882c855c64cbb8188d8a4d4da2ef9 2023-04-12 15:54:23,234 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/d5c882c855c64cbb8188d8a4d4da2ef9, entries=150, sequenceid=340, filesize=25.4 K 2023-04-12 15:54:23,238 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/B/0de51cfd163947f983b88582735cdeb1 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/0de51cfd163947f983b88582735cdeb1 2023-04-12 15:54:23,250 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/0de51cfd163947f983b88582735cdeb1, entries=150, sequenceid=340, filesize=11.9 K 2023-04-12 15:54:23,251 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/C/2e626e11a9c44828b4bdac944edba0d4 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/2e626e11a9c44828b4bdac944edba0d4 2023-04-12 15:54:23,257 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/2e626e11a9c44828b4bdac944edba0d4, entries=150, sequenceid=340, filesize=11.9 K 2023-04-12 15:54:23,258 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for ca92eda5f752c4c8f5e2f01f66bc73d8 in 1287ms, sequenceid=340, compaction requested=true 2023-04-12 15:54:23,258 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ca92eda5f752c4c8f5e2f01f66bc73d8: 2023-04-12 15:54:23,258 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:23,258 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:23,258 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:23,258 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:23,258 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:23,258 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:23,261 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:23,261 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:23,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:23,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:23,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:23,261 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:23,261 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:23,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:23,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:23,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:23,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:23,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:23,263 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:23,263 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28a9d45b[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:23,263 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:23,263 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:23,265 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:23,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:23,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:23,265 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:23,265 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:23,265 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:23,265 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:23,265 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:23,267 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:23,267 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:23,267 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:23,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:23,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:23,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:23,267 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:23,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:23,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:23,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:23,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:23,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:23,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:23,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:23,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:23,269 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:23,269 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:23,269 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:23,269 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:23,269 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:23,270 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:23,271 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:23,271 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:23,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:23,271 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:23,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:23,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:23,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:23,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:23,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:23,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:23,271 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:23,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:23,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:23,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:23,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:23,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:23,273 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:23,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:23,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:23,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:23,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:23,275 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:23,275 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:23,275 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:23,275 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:23,275 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:23,275 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:23,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:23,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:23,275 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:23,275 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:23,275 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:23,275 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:23,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:23,276 DEBUG [Thread-831] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:23,276 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:23,276 DEBUG [Thread-831] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:23,276 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:23,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:23,276 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:23,376 DEBUG [Thread-831] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:23,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:23,378 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:23,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:23,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:23,380 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:23,380 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:23,380 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:23,380 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:23,380 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:23,380 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:23,382 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:23,382 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:23,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:23,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:23,383 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:23,383 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:23,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:23,383 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:23,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:23,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:23,384 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:23,384 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:23,384 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:23,384 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:23,384 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:23,384 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:23,384 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:23,384 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:23,384 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:23,384 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. started... 2023-04-12 15:54:23,384 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ca92eda5f752c4c8f5e2f01f66bc73d8 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-12 15:54:23,385 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca92eda5f752c4c8f5e2f01f66bc73d8, store=A 2023-04-12 15:54:23,385 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:23,385 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca92eda5f752c4c8f5e2f01f66bc73d8, store=B 2023-04-12 15:54:23,385 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:23,385 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca92eda5f752c4c8f5e2f01f66bc73d8, store=C 2023-04-12 15:54:23,385 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:23,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on ca92eda5f752c4c8f5e2f01f66bc73d8 2023-04-12 15:54:23,475 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. as already flushing 2023-04-12 15:54:23,487 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:23,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34066 deadline: 1681314923484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:23,487 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:23,487 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:23,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34094 deadline: 1681314923484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:23,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34134 deadline: 1681314923485, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:23,592 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:23,593 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:23,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34066 deadline: 1681314923588, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:23,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34134 deadline: 1681314923588, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:23,593 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:23,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34094 deadline: 1681314923588, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:23,796 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:23,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34066 deadline: 1681314923794, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:23,797 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:23,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34134 deadline: 1681314923794, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:23,797 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:23,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34094 deadline: 1681314923795, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:23,800 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041207a3a1f6ae4b40d5895c2c553e6b1a68 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041207a3a1f6ae4b40d5895c2c553e6b1a68 2023-04-12 15:54:23,805 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=358, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/A/105356904e5f44af9215e2323dca6277 2023-04-12 15:54:23,815 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=358 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/B/f76cbe9988cf4572a5655f8b17bad887 2023-04-12 15:54:24,100 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:24,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34066 deadline: 1681314924097, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:24,102 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:24,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34094 deadline: 1681314924099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:24,103 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:24,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34134 deadline: 1681314924099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:24,225 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=358 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/C/8ff56a57419448128f72f8e54a9be6ef 2023-04-12 15:54:24,229 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/A/105356904e5f44af9215e2323dca6277 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/105356904e5f44af9215e2323dca6277 2023-04-12 15:54:24,233 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/105356904e5f44af9215e2323dca6277, entries=150, sequenceid=358, filesize=25.4 K 2023-04-12 15:54:24,234 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/B/f76cbe9988cf4572a5655f8b17bad887 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/f76cbe9988cf4572a5655f8b17bad887 2023-04-12 15:54:24,238 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/f76cbe9988cf4572a5655f8b17bad887, entries=150, sequenceid=358, filesize=11.9 K 2023-04-12 15:54:24,239 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/C/8ff56a57419448128f72f8e54a9be6ef as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/8ff56a57419448128f72f8e54a9be6ef 2023-04-12 15:54:24,244 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/8ff56a57419448128f72f8e54a9be6ef, entries=150, sequenceid=358, filesize=11.9 K 2023-04-12 15:54:24,244 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for ca92eda5f752c4c8f5e2f01f66bc73d8 in 860ms, sequenceid=358, compaction requested=true 2023-04-12 15:54:24,244 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ca92eda5f752c4c8f5e2f01f66bc73d8: 2023-04-12 15:54:24,244 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:24,244 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:24,245 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:24,245 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:24,245 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:24,245 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:24,246 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:24,246 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:24,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:24,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:24,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:24,247 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:24,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:24,247 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:24,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:24,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:24,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:24,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:24,248 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:24,248 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11d6061f[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:24,248 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:24,248 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:24,249 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:24,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:24,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:24,249 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:24,249 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:24,249 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:24,250 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:24,250 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:24,251 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:24,251 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:24,251 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:24,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:24,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:24,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:24,251 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:24,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:24,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:24,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:24,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:24,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:24,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:24,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:24,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:24,253 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:24,253 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:24,253 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:24,253 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:24,253 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:24,253 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:24,255 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:24,255 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:24,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:24,255 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:24,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:24,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:24,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:24,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:24,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:24,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:24,256 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:24,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:24,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:24,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:24,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:24,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:24,257 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:24,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:24,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:24,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:24,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:24,260 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:24,260 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:24,260 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:24,260 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:24,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:24,260 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:24,260 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:24,260 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:24,260 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:24,260 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:24,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:24,260 DEBUG [Thread-831] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:24,260 DEBUG [Thread-831] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:24,260 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:24,260 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:24,260 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:24,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:24,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:24,360 DEBUG [Thread-831] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:24,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:24,362 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:24,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:24,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:24,364 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:24,364 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:24,364 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:24,364 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:24,364 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:24,364 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:24,366 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:24,366 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:24,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:24,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:24,366 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:24,366 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:24,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:24,367 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:24,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:24,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:24,367 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:24,367 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:24,367 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:24,367 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:24,367 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:24,367 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:24,368 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:24,368 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:24,368 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:24,368 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. started... 2023-04-12 15:54:24,368 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ca92eda5f752c4c8f5e2f01f66bc73d8 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-12 15:54:24,368 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca92eda5f752c4c8f5e2f01f66bc73d8, store=A 2023-04-12 15:54:24,368 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:24,368 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca92eda5f752c4c8f5e2f01f66bc73d8, store=B 2023-04-12 15:54:24,368 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:24,368 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca92eda5f752c4c8f5e2f01f66bc73d8, store=C 2023-04-12 15:54:24,368 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:24,384 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230412a64c1b3e1ff94faeb25fed29c173652c to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412a64c1b3e1ff94faeb25fed29c173652c 2023-04-12 15:54:24,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on ca92eda5f752c4c8f5e2f01f66bc73d8 2023-04-12 15:54:24,602 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. as already flushing 2023-04-12 15:54:24,625 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:24,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34094 deadline: 1681314924622, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:24,626 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:24,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34134 deadline: 1681314924623, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:24,629 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:24,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34066 deadline: 1681314924625, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:24,727 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:24,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34094 deadline: 1681314924726, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:24,728 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:24,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34134 deadline: 1681314924727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:24,730 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:24,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34066 deadline: 1681314924729, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:24,790 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=372, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/A/f266e1b59bab4f34be489b507bb1145c 2023-04-12 15:54:24,930 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:24,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34094 deadline: 1681314924928, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:24,931 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:24,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34134 deadline: 1681314924930, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:24,933 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:24,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34066 deadline: 1681314924931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:25,202 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=372 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/B/65b43acc607c4ef2b4e287077707fce9 2023-04-12 15:54:25,232 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:25,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34094 deadline: 1681314925231, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:25,237 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:25,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34134 deadline: 1681314925233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:25,237 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:25,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34066 deadline: 1681314925235, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:25,254 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:25,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34122 deadline: 1681314925252, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:25,255 DEBUG [Thread-821] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8165 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8., hostname=jenkins-hbase4.apache.org,38905,1681314818739, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-12 15:54:25,615 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=372 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/C/3b8af9470c3644ce86ff6df3155798c1 2023-04-12 15:54:25,619 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/A/f266e1b59bab4f34be489b507bb1145c as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/f266e1b59bab4f34be489b507bb1145c 2023-04-12 15:54:25,623 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/f266e1b59bab4f34be489b507bb1145c, entries=150, sequenceid=372, filesize=25.4 K 2023-04-12 15:54:25,624 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/B/65b43acc607c4ef2b4e287077707fce9 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/65b43acc607c4ef2b4e287077707fce9 2023-04-12 15:54:25,628 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/65b43acc607c4ef2b4e287077707fce9, entries=150, sequenceid=372, filesize=11.9 K 2023-04-12 15:54:25,629 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/C/3b8af9470c3644ce86ff6df3155798c1 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/3b8af9470c3644ce86ff6df3155798c1 2023-04-12 15:54:25,633 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/3b8af9470c3644ce86ff6df3155798c1, entries=150, sequenceid=372, filesize=11.9 K 2023-04-12 15:54:25,633 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for ca92eda5f752c4c8f5e2f01f66bc73d8 in 1265ms, sequenceid=372, compaction requested=true 2023-04-12 15:54:25,634 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ca92eda5f752c4c8f5e2f01f66bc73d8: 2023-04-12 15:54:25,634 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:25,634 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:25,634 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:25,634 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:25,634 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:25,634 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:25,636 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:25,636 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:25,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:25,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:25,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:25,636 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:25,636 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:25,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:25,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:25,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:25,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:25,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:25,637 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:25,637 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b4cdd1c[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:25,637 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:25,638 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:25,640 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:25,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:25,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:25,640 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:25,640 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:25,640 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:25,640 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:25,640 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:25,641 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:25,641 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:25,641 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:25,641 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:25,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:25,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:25,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:25,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:25,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:25,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:25,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:25,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:25,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:25,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:25,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:25,644 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:25,644 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:25,644 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:25,644 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:25,644 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:25,644 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:25,646 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:25,646 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:25,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:25,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:25,646 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:25,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:25,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:25,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:25,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:25,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:25,646 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:25,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:25,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:25,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:25,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:25,647 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:25,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:25,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:25,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:25,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:25,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:25,653 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:25,653 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:25,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:25,653 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:25,653 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:25,653 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:25,653 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:25,653 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:25,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:25,653 DEBUG [Thread-831] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:25,654 DEBUG [Thread-831] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:25,653 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:25,654 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:25,654 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:25,654 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:25,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:25,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:25,654 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:25,654 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:25,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on ca92eda5f752c4c8f5e2f01f66bc73d8 2023-04-12 15:54:25,737 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ca92eda5f752c4c8f5e2f01f66bc73d8 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-12 15:54:25,738 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca92eda5f752c4c8f5e2f01f66bc73d8, store=A 2023-04-12 15:54:25,738 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:25,738 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca92eda5f752c4c8f5e2f01f66bc73d8, store=B 2023-04-12 15:54:25,738 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:25,738 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca92eda5f752c4c8f5e2f01f66bc73d8, store=C 2023-04-12 15:54:25,738 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:25,754 DEBUG [Thread-831] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:25,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:25,755 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:25,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:25,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:25,757 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:25,757 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:25,757 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:25,757 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:25,757 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:25,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34134 deadline: 1681314925751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:25,757 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:25,757 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:25,758 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:25,758 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:25,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:25,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:25,759 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:25,759 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:25,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:25,759 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:25,759 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:25,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:25,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:25,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34094 deadline: 1681314925752, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:25,759 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:25,761 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:25,761 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:25,761 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:25,761 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:25,761 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:25,761 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:25,761 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:25,761 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:25,761 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. started... 2023-04-12 15:54:25,761 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. as already flushing 2023-04-12 15:54:25,761 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:25,761 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:25,761 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:25,761 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:25,761 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:25,761 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:25,761 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:25,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34066 deadline: 1681314925757, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:25,763 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:25,763 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:25,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:25,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:25,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:25,763 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:25,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:25,763 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:25,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:25,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:25,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:25,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:25,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:25,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@602324c7[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:25,765 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:25,765 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:25,766 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:25,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:25,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:25,766 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:25,766 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:25,766 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:25,766 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:25,766 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:25,768 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:25,768 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:25,768 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:25,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:25,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:25,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:25,768 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:25,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:25,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:25,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:25,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:25,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:25,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:25,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:25,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:25,770 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:25,770 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:25,771 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:25,771 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:25,771 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:25,771 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:25,772 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:25,772 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:25,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:25,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:25,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:25,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:25,772 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:25,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:25,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:25,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:25,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:25,773 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:25,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:25,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:25,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:25,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:25,774 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:25,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:25,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:25,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:25,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:25,776 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:25,776 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:25,776 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:25,776 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:25,776 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:25,776 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:25,776 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:25,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:25,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:25,777 DEBUG [Thread-831] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:25,777 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:25,776 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:25,776 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:25,776 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:25,777 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:25,777 DEBUG [Thread-831] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:25,777 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:25,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:25,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:25,861 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:25,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34134 deadline: 1681314925858, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:25,862 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:25,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34094 deadline: 1681314925860, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:25,865 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:25,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34066 deadline: 1681314925862, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:25,877 DEBUG [Thread-831] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:25,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:25,879 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:25,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:25,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:25,880 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:25,880 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:25,880 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:25,880 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:25,881 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:25,881 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:25,882 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:25,882 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:25,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:25,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:25,883 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:25,883 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:25,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:25,883 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:25,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:25,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:25,883 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:25,884 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:25,884 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:25,884 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:25,884 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:25,884 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:25,884 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:25,884 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:25,884 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:25,884 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. started... 2023-04-12 15:54:25,884 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. as already flushing 2023-04-12 15:54:25,884 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:25,884 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:25,884 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:25,884 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:25,884 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:25,884 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:25,887 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:25,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:25,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:25,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:25,887 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:25,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:25,888 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:25,888 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:25,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:25,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:25,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:25,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:25,889 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:25,889 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ce2f94[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:25,889 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:25,889 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:25,890 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:25,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:25,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:25,890 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:25,890 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:25,890 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:25,890 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:25,890 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:25,891 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:25,892 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:25,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:25,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:25,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:25,892 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:25,892 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:25,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:25,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:25,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:25,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:25,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:25,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:25,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:25,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:25,894 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:25,894 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:25,894 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:25,894 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:25,894 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:25,894 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:25,896 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:25,896 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:25,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:25,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:25,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:25,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:25,896 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:25,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:25,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:25,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:25,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:25,897 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:25,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:25,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:25,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:25,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:25,898 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:25,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:25,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:25,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:25,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:25,901 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:25,901 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:25,901 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:25,901 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:25,901 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:25,901 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:25,901 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:25,902 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:25,901 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:25,902 DEBUG [Thread-831] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:25,902 DEBUG [Thread-831] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:25,902 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:25,902 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:25,902 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:25,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:25,902 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:25,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:25,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:25,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:26,002 DEBUG [Thread-831] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:26,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:26,004 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:26,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:26,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:26,005 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:26,005 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:26,005 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:26,005 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:26,006 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:26,006 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:26,007 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:26,007 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:26,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:26,007 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,007 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:26,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:26,008 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:26,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:26,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:26,008 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:26,008 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:26,008 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:26,008 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:26,008 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:26,008 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:26,008 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:26,008 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:26,008 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:26,009 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. started... 2023-04-12 15:54:26,009 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. as already flushing 2023-04-12 15:54:26,009 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:26,009 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:26,009 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:26,009 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,009 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:26,009 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:26,010 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,010 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:26,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:26,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:26,010 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:26,010 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:26,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:26,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:26,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:26,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:26,012 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:26,012 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@278927ac[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:26,012 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:26,012 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:26,013 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:26,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:26,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:26,013 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:26,013 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,013 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:26,013 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:26,013 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:26,015 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:26,015 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,015 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:26,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:26,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:26,016 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:26,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:26,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:26,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:26,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:26,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:26,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:26,018 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:26,018 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:26,018 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:26,018 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:26,018 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:26,018 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:26,019 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:26,019 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:26,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:26,019 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:26,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:26,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:26,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:26,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:26,020 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:26,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:26,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:26,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:26,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:26,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:26,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:26,025 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:26,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:26,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,029 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,029 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:26,029 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:26,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:26,030 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:26,029 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:26,030 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:26,030 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:26,030 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:26,030 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:26,030 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:26,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:26,030 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:26,030 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:26,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:26,030 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:26,031 DEBUG [Thread-831] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:26,031 DEBUG [Thread-831] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:26,063 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:26,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34094 deadline: 1681314926063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,063 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:26,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34134 deadline: 1681314926063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,068 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:26,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34066 deadline: 1681314926067, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,131 DEBUG [Thread-831] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:26,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:26,133 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:26,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:26,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:26,134 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:26,134 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:26,134 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:26,134 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:26,135 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:26,135 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:26,137 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:26,137 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:26,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:26,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:26,137 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,137 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:26,137 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:26,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:26,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:26,138 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:26,138 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:26,138 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:26,138 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:26,138 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:26,138 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:26,139 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:26,139 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:26,139 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:26,139 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. started... 2023-04-12 15:54:26,139 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. as already flushing 2023-04-12 15:54:26,139 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:26,139 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:26,139 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:26,139 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,139 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:26,139 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:26,140 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,140 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:26,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:26,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:26,141 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:26,141 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:26,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:26,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:26,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:26,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:26,142 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:26,142 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ac0ba7c[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:26,142 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:26,142 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:26,143 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:26,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:26,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:26,143 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:26,143 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:26,143 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,144 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:26,143 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:26,145 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,145 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:26,145 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:26,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:26,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:26,145 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:26,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:26,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:26,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:26,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:26,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:26,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:26,148 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:26,148 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:26,148 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:26,148 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:26,148 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:26,148 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:26,150 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:26,150 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:26,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:26,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:26,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:26,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:26,150 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:26,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:26,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:26,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:26,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:26,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:26,151 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304121040e86e256e44e681d6f44283e81506 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304121040e86e256e44e681d6f44283e81506 2023-04-12 15:54:26,151 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:26,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:26,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:26,154 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:26,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:26,157 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:26,157 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,157 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:26,157 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:26,157 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:26,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:26,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:26,157 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:26,157 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:26,157 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,157 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:26,157 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:26,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:26,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:26,157 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:26,157 DEBUG [Thread-831] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:26,158 DEBUG [Thread-831] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:26,158 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:26,158 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:26,258 DEBUG [Thread-831] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:26,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:26,260 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:26,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:26,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:26,261 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:26,261 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:26,261 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:26,261 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:26,262 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:26,262 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:26,263 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,263 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:26,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:26,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:26,263 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,263 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:26,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:26,264 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:26,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:26,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:26,264 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:26,264 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:26,264 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:26,264 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:26,265 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:26,265 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:26,265 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:26,265 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:26,265 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:26,265 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. started... 2023-04-12 15:54:26,265 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. as already flushing 2023-04-12 15:54:26,265 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:26,265 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:26,265 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:26,265 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,265 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:26,265 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:26,266 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,266 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:26,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:26,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:26,267 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:26,267 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:26,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:26,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:26,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:26,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:26,268 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:26,268 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33bcbaff[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:26,268 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:26,268 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:26,271 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:26,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:26,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:26,271 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:26,271 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,271 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:26,271 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:26,271 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:26,272 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,272 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:26,273 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:26,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,273 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:26,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:26,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:26,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:26,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:26,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:26,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:26,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:26,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:26,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:26,275 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:26,275 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:26,275 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:26,275 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:26,275 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:26,276 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:26,277 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:26,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:26,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:26,277 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:26,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:26,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:26,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:26,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:26,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:26,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:26,277 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:26,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:26,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:26,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:26,278 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:26,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:26,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,280 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:26,281 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:26,281 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,281 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:26,281 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:26,281 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:26,281 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:26,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:26,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:26,281 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,281 DEBUG [Thread-831] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:26,281 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:26,281 DEBUG [Thread-831] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:26,281 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:26,281 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:26,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:26,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:26,281 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:26,281 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:26,365 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:26,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34094 deadline: 1681314926364, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,365 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:26,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34134 deadline: 1681314926364, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,373 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:26,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34066 deadline: 1681314926371, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,381 DEBUG [Thread-831] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:26,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:26,383 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:26,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:26,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:26,384 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:26,384 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:26,384 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:26,384 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:26,385 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:26,385 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:26,387 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:26,387 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:26,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:26,387 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,387 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:26,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:26,388 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:26,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:26,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:26,388 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:26,388 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:26,388 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:26,388 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:26,388 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:26,388 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:26,389 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:26,389 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:26,389 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:26,389 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. started... 2023-04-12 15:54:26,389 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. as already flushing 2023-04-12 15:54:26,389 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:26,389 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:26,389 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:26,389 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,389 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:26,389 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:26,390 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,390 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:26,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:26,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:26,391 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:26,391 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:26,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:26,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:26,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:26,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:26,392 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:26,392 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44f99193[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:26,392 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:26,392 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:26,393 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:26,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:26,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:26,394 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:26,394 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:26,394 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,394 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:26,394 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:26,395 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,395 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:26,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:26,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:26,395 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:26,395 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:26,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:26,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:26,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:26,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:26,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:26,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:26,397 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:26,397 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:26,397 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:26,397 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:26,397 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:26,397 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:26,399 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:26,399 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:26,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:26,399 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:26,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:26,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:26,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:26,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:26,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:26,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:26,400 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:26,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:26,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:26,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:26,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:26,401 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:26,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:26,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,403 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,403 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:26,403 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:26,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:26,403 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:26,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:26,403 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:26,403 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,403 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:26,403 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:26,403 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:26,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:26,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:26,404 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:26,404 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:26,404 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:26,404 DEBUG [Thread-831] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:26,404 DEBUG [Thread-831] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:26,504 DEBUG [Thread-831] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:26,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:26,506 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:26,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:26,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:26,507 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:26,507 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:26,507 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:26,507 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:26,508 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:26,508 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:26,509 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:26,509 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:26,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:26,509 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,509 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:26,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:26,510 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:26,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:26,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:26,510 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:26,511 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:26,511 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:26,511 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:26,511 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:26,512 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:26,512 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:26,512 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:26,512 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:26,512 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. started... 2023-04-12 15:54:26,512 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. as already flushing 2023-04-12 15:54:26,512 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:26,512 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:26,512 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:26,512 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,512 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:26,512 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:26,519 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:26,519 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,520 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:26,520 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:26,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:26,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:26,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:26,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:26,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:26,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:26,521 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:26,521 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:26,521 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a236ae7[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:26,521 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:26,522 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:26,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:26,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:26,522 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:26,522 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:26,523 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:26,523 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,523 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:26,524 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:26,524 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,524 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:26,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:26,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:26,524 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:26,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:26,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:26,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:26,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:26,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:26,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:26,527 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:26,527 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:26,527 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:26,527 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:26,527 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:26,527 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:26,530 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:26,530 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:26,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:26,530 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:26,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:26,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:26,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:26,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:26,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:26,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:26,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:26,531 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:26,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:26,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:26,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:26,534 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:26,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:26,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,538 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:26,538 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:26,539 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:26,539 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,539 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:26,539 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:26,539 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:26,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:26,539 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:26,539 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,539 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:26,539 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:26,539 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:26,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:26,539 DEBUG [Thread-831] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:26,539 DEBUG [Thread-831] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:26,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:26,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:26,540 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:26,561 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=391, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/A/a0545103cb2c44cda1839f60a7980166 2023-04-12 15:54:26,574 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=391 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/B/8e672c6e6ce34a2592a808228e2504de 2023-04-12 15:54:26,639 DEBUG [Thread-831] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:26,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:26,641 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:26,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:26,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:26,643 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:26,643 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:26,643 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:26,643 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:26,643 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:26,643 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:26,645 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:26,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:26,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:26,645 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:26,645 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,645 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:26,645 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:26,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:26,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:26,646 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:26,646 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:26,646 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:26,646 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:26,646 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:26,646 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:26,646 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:26,646 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:26,646 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:26,646 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. started... 2023-04-12 15:54:26,647 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. as already flushing 2023-04-12 15:54:26,647 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:26,647 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:26,647 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:26,647 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,647 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:26,647 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:26,648 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,648 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:26,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:26,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:26,649 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:26,649 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:26,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:26,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:26,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:26,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:26,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:26,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65ff2685[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:26,650 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:26,650 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:26,651 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:26,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:26,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:26,651 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:26,651 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:26,651 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:26,651 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,652 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:26,657 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,657 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:26,657 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:26,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:26,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:26,657 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:26,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:26,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:26,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:26,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:26,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:26,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:26,659 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:26,659 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:26,659 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:26,659 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:26,659 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:26,659 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:26,661 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:26,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:26,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:26,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:26,661 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:26,661 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:26,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:26,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:26,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:26,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:26,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:26,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:26,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:26,666 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:26,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:26,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,667 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:26,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:26,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,670 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:26,670 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:26,670 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,670 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:26,670 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:26,670 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:26,670 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:26,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:26,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:26,670 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,670 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:26,670 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:26,670 DEBUG [Thread-831] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:26,670 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:26,670 DEBUG [Thread-831] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:26,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:26,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:26,670 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:26,771 DEBUG [Thread-831] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:26,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:26,773 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:26,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:26,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:26,775 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:26,775 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:26,775 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:26,775 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:26,775 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:26,775 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:26,776 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:26,776 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:26,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:26,779 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,779 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:26,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:26,779 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:26,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:26,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:26,780 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:26,780 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:26,780 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:26,780 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:26,780 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:26,780 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:26,780 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:26,780 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:26,780 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:26,780 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. started... 2023-04-12 15:54:26,781 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. as already flushing 2023-04-12 15:54:26,781 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:26,781 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:26,781 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:26,781 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,781 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:26,781 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:26,784 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,784 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:26,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:26,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:26,784 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:26,784 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:26,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:26,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:26,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:26,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:26,786 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:26,786 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33c21387[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:26,786 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:26,787 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:26,788 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:26,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:26,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:26,788 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:26,788 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:26,788 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,788 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:26,788 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:26,789 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:26,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:26,790 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:26,790 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:26,790 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:26,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:26,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:26,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:26,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:26,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:26,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:26,792 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:26,792 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:26,792 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:26,792 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:26,792 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:26,792 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:26,794 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:26,794 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:26,794 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:26,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:26,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:26,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:26,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:26,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:26,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:26,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:26,794 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:26,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:26,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:26,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:26,796 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:26,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:26,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:26,799 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,799 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:26,799 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:26,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:26,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:26,799 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:26,799 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:26,799 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:26,799 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:26,799 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:26,799 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,799 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:26,799 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:26,799 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:26,799 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:26,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:26,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:26,799 DEBUG [Thread-831] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:26,800 DEBUG [Thread-831] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:26,869 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:26,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34134 deadline: 1681314926867, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,870 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:26,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34094 deadline: 1681314926868, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,879 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:26,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34066 deadline: 1681314926875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,900 DEBUG [Thread-831] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:26,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:26,902 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:26,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:26,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:26,903 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:26,903 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:26,903 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:26,903 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:26,904 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:26,904 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:26,905 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:26,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:26,905 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:26,905 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,905 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:26,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:26,906 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:26,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:26,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:26,906 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:26,906 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:26,906 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:26,906 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:26,906 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:26,907 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:26,907 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:26,907 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:26,907 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:26,907 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. started... 2023-04-12 15:54:26,907 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. as already flushing 2023-04-12 15:54:26,907 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:26,907 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:26,907 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:26,907 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,907 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:26,907 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:26,908 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,908 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:26,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:26,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:26,909 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:26,909 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:26,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:26,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:26,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:26,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:26,910 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:26,910 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b3f7965[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:26,910 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:26,910 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:26,915 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:26,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:26,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:26,916 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:26,916 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:26,916 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:26,916 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,916 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:26,917 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:26,917 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,917 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:26,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:26,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:26,917 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:26,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:26,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:26,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:26,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:26,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:26,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:26,920 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:26,920 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:26,920 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:26,920 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:26,920 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:26,920 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:26,922 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:26,922 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:26,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:26,922 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:26,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:26,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:26,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:26,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:26,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:26,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:26,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:26,922 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:26,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:26,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:26,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:26,923 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:26,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:26,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,925 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,925 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:26,925 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:26,925 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:26,926 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:26,925 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:26,925 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:26,925 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:26,926 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:26,926 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:26,926 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:26,925 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:26,926 DEBUG [Thread-831] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:26,926 DEBUG [Thread-831] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:26,926 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:26,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:26,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:26,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:26,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:26,989 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=391 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/C/6e75b143a8f344beab12241bcb11ee71 2023-04-12 15:54:26,994 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/A/a0545103cb2c44cda1839f60a7980166 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/a0545103cb2c44cda1839f60a7980166 2023-04-12 15:54:26,999 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/a0545103cb2c44cda1839f60a7980166, entries=200, sequenceid=391, filesize=32.3 K 2023-04-12 15:54:26,999 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/B/8e672c6e6ce34a2592a808228e2504de as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/8e672c6e6ce34a2592a808228e2504de 2023-04-12 15:54:27,003 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/8e672c6e6ce34a2592a808228e2504de, entries=150, sequenceid=391, filesize=11.9 K 2023-04-12 15:54:27,007 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/C/6e75b143a8f344beab12241bcb11ee71 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/6e75b143a8f344beab12241bcb11ee71 2023-04-12 15:54:27,019 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/6e75b143a8f344beab12241bcb11ee71, entries=150, sequenceid=391, filesize=11.9 K 2023-04-12 15:54:27,020 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for ca92eda5f752c4c8f5e2f01f66bc73d8 in 1283ms, sequenceid=391, compaction requested=true 2023-04-12 15:54:27,020 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ca92eda5f752c4c8f5e2f01f66bc73d8: 2023-04-12 15:54:27,020 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:54:27,020 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-12 15:54:27,020 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-12 15:54:27,020 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:54:27,020 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:54:27,022 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 163930 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-12 15:54:27,022 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1912): ca92eda5f752c4c8f5e2f01f66bc73d8/A is initiating minor compaction (all files) 2023-04-12 15:54:27,023 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ca92eda5f752c4c8f5e2f01f66bc73d8/A in TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:27,023 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/ca2d4af540b24fb38a0fa858ddbd6042, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/720dfb88249b4815bd19e329e3c100cb, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/d5c882c855c64cbb8188d8a4d4da2ef9, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/105356904e5f44af9215e2323dca6277, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/f266e1b59bab4f34be489b507bb1145c, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/a0545103cb2c44cda1839f60a7980166] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp, totalSize=160.1 K 2023-04-12 15:54:27,023 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73730 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-12 15:54:27,023 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting ca2d4af540b24fb38a0fa858ddbd6042, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=303, earliestPutTs=1681314859474 2023-04-12 15:54:27,023 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1912): ca92eda5f752c4c8f5e2f01f66bc73d8/B is initiating minor compaction (all files) 2023-04-12 15:54:27,023 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ca92eda5f752c4c8f5e2f01f66bc73d8/B in TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:27,023 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/8b50d0c3dcdb423d92caa6e6db2b61c9, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/5ac59dfdb9d44efd8f790ec7d0631ca1, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/0de51cfd163947f983b88582735cdeb1, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/f76cbe9988cf4572a5655f8b17bad887, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/65b43acc607c4ef2b4e287077707fce9, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/8e672c6e6ce34a2592a808228e2504de] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp, totalSize=72.0 K 2023-04-12 15:54:27,024 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 720dfb88249b4815bd19e329e3c100cb, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=323, earliestPutTs=1681314859596 2023-04-12 15:54:27,024 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 8b50d0c3dcdb423d92caa6e6db2b61c9, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=303, earliestPutTs=1681314859474 2023-04-12 15:54:27,024 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting d5c882c855c64cbb8188d8a4d4da2ef9, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=340, earliestPutTs=1681314860222 2023-04-12 15:54:27,024 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 5ac59dfdb9d44efd8f790ec7d0631ca1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=323, earliestPutTs=1681314859596 2023-04-12 15:54:27,025 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 105356904e5f44af9215e2323dca6277, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=358, earliestPutTs=1681314862356 2023-04-12 15:54:27,025 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 0de51cfd163947f983b88582735cdeb1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=340, earliestPutTs=1681314860222 2023-04-12 15:54:27,025 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting f266e1b59bab4f34be489b507bb1145c, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=372, earliestPutTs=1681314863476 2023-04-12 15:54:27,025 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting f76cbe9988cf4572a5655f8b17bad887, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=358, earliestPutTs=1681314862356 2023-04-12 15:54:27,025 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting a0545103cb2c44cda1839f60a7980166, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=391, earliestPutTs=1681314864616 2023-04-12 15:54:27,026 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 65b43acc607c4ef2b4e287077707fce9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=372, earliestPutTs=1681314863476 2023-04-12 15:54:27,026 DEBUG [Thread-831] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:27,026 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 8e672c6e6ce34a2592a808228e2504de, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=391, earliestPutTs=1681314864616 2023-04-12 15:54:27,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:27,030 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:27,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:27,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:27,032 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:27,032 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:27,032 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:27,032 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:27,032 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:27,032 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:27,035 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:27,035 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:27,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:27,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:27,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:27,036 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:27,036 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:27,036 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:27,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:27,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:27,037 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:27,044 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:27,044 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:27,044 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:27,044 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:27,044 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:27,047 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] throttle.PressureAwareThroughputController(145): ca92eda5f752c4c8f5e2f01f66bc73d8#B#compaction#174 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:54:27,052 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:27,052 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:27,052 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:27,052 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. started... 2023-04-12 15:54:27,052 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ca92eda5f752c4c8f5e2f01f66bc73d8 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-12 15:54:27,052 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca92eda5f752c4c8f5e2f01f66bc73d8, store=A 2023-04-12 15:54:27,052 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:27,052 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca92eda5f752c4c8f5e2f01f66bc73d8, store=B 2023-04-12 15:54:27,053 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:27,053 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca92eda5f752c4c8f5e2f01f66bc73d8, store=C 2023-04-12 15:54:27,053 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:27,074 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] throttle.PressureAwareThroughputController(145): ca92eda5f752c4c8f5e2f01f66bc73d8#A#compaction#175 average throughput is 0.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:54:27,090 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230412366f143f618248b0997f9b783d4b800e to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412366f143f618248b0997f9b783d4b800e 2023-04-12 15:54:27,106 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=404, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/A/74cdd999b19b48978f5ad24bab1c1ff8 2023-04-12 15:54:27,479 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/B/e63b50ac824b437f9e2f8e02d82948d7 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/e63b50ac824b437f9e2f8e02d82948d7 2023-04-12 15:54:27,487 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in ca92eda5f752c4c8f5e2f01f66bc73d8/B of ca92eda5f752c4c8f5e2f01f66bc73d8 into e63b50ac824b437f9e2f8e02d82948d7(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:54:27,487 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ca92eda5f752c4c8f5e2f01f66bc73d8: 2023-04-12 15:54:27,487 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8., storeName=ca92eda5f752c4c8f5e2f01f66bc73d8/B, priority=10, startTime=1681314867020; duration=0sec 2023-04-12 15:54:27,487 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:54:27,487 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-12 15:54:27,490 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73730 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-12 15:54:27,490 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1912): ca92eda5f752c4c8f5e2f01f66bc73d8/C is initiating minor compaction (all files) 2023-04-12 15:54:27,490 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ca92eda5f752c4c8f5e2f01f66bc73d8/C in TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:27,490 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/c2f49a8f5d6d44558e087af982d82ad8, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/49ff5b64306148729f8bdbe9ddc846f9, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/2e626e11a9c44828b4bdac944edba0d4, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/8ff56a57419448128f72f8e54a9be6ef, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/3b8af9470c3644ce86ff6df3155798c1, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/6e75b143a8f344beab12241bcb11ee71] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp, totalSize=72.0 K 2023-04-12 15:54:27,491 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting c2f49a8f5d6d44558e087af982d82ad8, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=303, earliestPutTs=1681314859474 2023-04-12 15:54:27,492 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 49ff5b64306148729f8bdbe9ddc846f9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=323, earliestPutTs=1681314859596 2023-04-12 15:54:27,492 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 2e626e11a9c44828b4bdac944edba0d4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=340, earliestPutTs=1681314860222 2023-04-12 15:54:27,493 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 8ff56a57419448128f72f8e54a9be6ef, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=358, earliestPutTs=1681314862356 2023-04-12 15:54:27,493 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 3b8af9470c3644ce86ff6df3155798c1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=372, earliestPutTs=1681314863476 2023-04-12 15:54:27,493 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 6e75b143a8f344beab12241bcb11ee71, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=391, earliestPutTs=1681314864616 2023-04-12 15:54:27,502 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/A/da4db90529f14c72bb4c16780e111929 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/da4db90529f14c72bb4c16780e111929 2023-04-12 15:54:27,509 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] throttle.PressureAwareThroughputController(145): ca92eda5f752c4c8f5e2f01f66bc73d8#C#compaction#178 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:54:27,509 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in ca92eda5f752c4c8f5e2f01f66bc73d8/A of ca92eda5f752c4c8f5e2f01f66bc73d8 into da4db90529f14c72bb4c16780e111929(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:54:27,509 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ca92eda5f752c4c8f5e2f01f66bc73d8: 2023-04-12 15:54:27,509 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8., storeName=ca92eda5f752c4c8f5e2f01f66bc73d8/A, priority=10, startTime=1681314867020; duration=0sec 2023-04-12 15:54:27,509 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:54:27,528 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=404 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/B/f149ccb0b7d44a7fad81f2d31dc61c73 2023-04-12 15:54:27,538 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/C/af2631c32ea54ed3b42c89ff2e36a03e as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/af2631c32ea54ed3b42c89ff2e36a03e 2023-04-12 15:54:27,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on ca92eda5f752c4c8f5e2f01f66bc73d8 2023-04-12 15:54:27,540 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. as already flushing 2023-04-12 15:54:27,546 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in ca92eda5f752c4c8f5e2f01f66bc73d8/C of ca92eda5f752c4c8f5e2f01f66bc73d8 into af2631c32ea54ed3b42c89ff2e36a03e(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:54:27,546 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ca92eda5f752c4c8f5e2f01f66bc73d8: 2023-04-12 15:54:27,546 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8., storeName=ca92eda5f752c4c8f5e2f01f66bc73d8/C, priority=10, startTime=1681314867020; duration=0sec 2023-04-12 15:54:27,546 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:54:27,555 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=404 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/C/78a5a72a6f48485b86d486c5ddacb09e 2023-04-12 15:54:27,560 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/A/74cdd999b19b48978f5ad24bab1c1ff8 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/74cdd999b19b48978f5ad24bab1c1ff8 2023-04-12 15:54:27,566 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/74cdd999b19b48978f5ad24bab1c1ff8, entries=150, sequenceid=404, filesize=25.4 K 2023-04-12 15:54:27,567 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/B/f149ccb0b7d44a7fad81f2d31dc61c73 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/f149ccb0b7d44a7fad81f2d31dc61c73 2023-04-12 15:54:27,572 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/f149ccb0b7d44a7fad81f2d31dc61c73, entries=150, sequenceid=404, filesize=11.9 K 2023-04-12 15:54:27,572 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/C/78a5a72a6f48485b86d486c5ddacb09e as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/78a5a72a6f48485b86d486c5ddacb09e 2023-04-12 15:54:27,578 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/78a5a72a6f48485b86d486c5ddacb09e, entries=150, sequenceid=404, filesize=11.9 K 2023-04-12 15:54:27,579 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=60.38 KB/61830 for ca92eda5f752c4c8f5e2f01f66bc73d8 in 527ms, sequenceid=404, compaction requested=false 2023-04-12 15:54:27,579 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ca92eda5f752c4c8f5e2f01f66bc73d8: 2023-04-12 15:54:27,579 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:27,579 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:27,579 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:27,579 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:27,579 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:27,579 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:27,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on ca92eda5f752c4c8f5e2f01f66bc73d8 2023-04-12 15:54:27,581 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ca92eda5f752c4c8f5e2f01f66bc73d8 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-12 15:54:27,581 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:27,581 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca92eda5f752c4c8f5e2f01f66bc73d8, store=A 2023-04-12 15:54:27,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:27,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:27,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:27,581 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:27,582 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:27,582 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca92eda5f752c4c8f5e2f01f66bc73d8, store=B 2023-04-12 15:54:27,582 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:27,582 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca92eda5f752c4c8f5e2f01f66bc73d8, store=C 2023-04-12 15:54:27,582 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:27,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:27,582 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:27,582 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:27,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:27,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:27,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:27,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:27,585 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:27,585 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a7a58b7[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:27,585 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:27,585 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:27,588 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:27,588 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:27,588 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:27,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:27,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:27,588 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:27,588 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:27,588 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:27,590 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:27,590 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:27,590 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:27,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:27,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:27,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:27,590 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:27,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:27,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:27,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:27,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:27,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:27,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:27,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:27,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:27,596 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:27,596 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:27,596 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:27,596 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:27,596 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:27,596 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:27,600 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:27,600 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:27,600 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:27,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:27,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:27,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:27,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:27,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:27,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:27,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:27,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:27,601 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:27,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:27,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:27,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:27,602 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:27,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:27,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:27,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:27,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:27,606 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:27,606 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:27,606 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:27,606 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:27,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:27,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:27,606 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:27,606 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:27,606 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:27,606 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:27,606 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:27,606 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:27,606 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:27,606 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:27,607 DEBUG [Thread-831] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:27,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:27,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:27,607 DEBUG [Thread-831] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:27,607 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:27,609 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230412d6ab4d6f7bbd4ae8b726d5f9ec89d834 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412d6ab4d6f7bbd4ae8b726d5f9ec89d834 2023-04-12 15:54:27,624 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=420, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/A/2b09ca7692ff4b329ed90a266898de5a 2023-04-12 15:54:27,638 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:27,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34118 deadline: 1681314927638, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:27,655 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=420 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/B/3668822b22f6428c8143d722c7504c01 2023-04-12 15:54:27,677 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=420 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/C/c00dea6e6e49410b86fb135e07cebf79 2023-04-12 15:54:27,683 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/A/2b09ca7692ff4b329ed90a266898de5a as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/2b09ca7692ff4b329ed90a266898de5a 2023-04-12 15:54:27,688 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/2b09ca7692ff4b329ed90a266898de5a, entries=200, sequenceid=420, filesize=32.3 K 2023-04-12 15:54:27,690 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/B/3668822b22f6428c8143d722c7504c01 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/3668822b22f6428c8143d722c7504c01 2023-04-12 15:54:27,695 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/3668822b22f6428c8143d722c7504c01, entries=150, sequenceid=420, filesize=11.9 K 2023-04-12 15:54:27,697 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/C/c00dea6e6e49410b86fb135e07cebf79 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/c00dea6e6e49410b86fb135e07cebf79 2023-04-12 15:54:27,700 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/c00dea6e6e49410b86fb135e07cebf79, entries=150, sequenceid=420, filesize=11.9 K 2023-04-12 15:54:27,702 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for ca92eda5f752c4c8f5e2f01f66bc73d8 in 121ms, sequenceid=420, compaction requested=true 2023-04-12 15:54:27,702 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ca92eda5f752c4c8f5e2f01f66bc73d8: 2023-04-12 15:54:27,702 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:54:27,702 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:54:27,702 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 15:54:27,702 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:54:27,702 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 15:54:27,703 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85996 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 15:54:27,703 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37376 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 15:54:27,703 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1912): ca92eda5f752c4c8f5e2f01f66bc73d8/A is initiating minor compaction (all files) 2023-04-12 15:54:27,704 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1912): ca92eda5f752c4c8f5e2f01f66bc73d8/B is initiating minor compaction (all files) 2023-04-12 15:54:27,704 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ca92eda5f752c4c8f5e2f01f66bc73d8/A in TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:27,704 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ca92eda5f752c4c8f5e2f01f66bc73d8/B in TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:27,704 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/da4db90529f14c72bb4c16780e111929, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/74cdd999b19b48978f5ad24bab1c1ff8, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/2b09ca7692ff4b329ed90a266898de5a] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp, totalSize=84.0 K 2023-04-12 15:54:27,704 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/e63b50ac824b437f9e2f8e02d82948d7, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/f149ccb0b7d44a7fad81f2d31dc61c73, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/3668822b22f6428c8143d722c7504c01] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp, totalSize=36.5 K 2023-04-12 15:54:27,704 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting da4db90529f14c72bb4c16780e111929, keycount=150, bloomtype=ROW, size=26.2 K, encoding=NONE, compression=NONE, seqNum=391, earliestPutTs=1681314864616 2023-04-12 15:54:27,704 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting e63b50ac824b437f9e2f8e02d82948d7, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=391, earliestPutTs=1681314864616 2023-04-12 15:54:27,705 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 74cdd999b19b48978f5ad24bab1c1ff8, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=404, earliestPutTs=1681314865748 2023-04-12 15:54:27,705 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting f149ccb0b7d44a7fad81f2d31dc61c73, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=404, earliestPutTs=1681314865748 2023-04-12 15:54:27,705 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 2b09ca7692ff4b329ed90a266898de5a, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=420, earliestPutTs=1681314867561 2023-04-12 15:54:27,705 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 3668822b22f6428c8143d722c7504c01, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=420, earliestPutTs=1681314867561 2023-04-12 15:54:27,707 DEBUG [Thread-831] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:27,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:27,710 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:27,718 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] throttle.PressureAwareThroughputController(145): ca92eda5f752c4c8f5e2f01f66bc73d8#B#compaction#184 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:54:27,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:27,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:27,719 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:27,719 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:27,719 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:27,719 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:27,721 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:27,721 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:27,722 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:27,722 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:27,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:27,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:27,723 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:27,723 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:27,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:27,723 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:27,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:27,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:27,724 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:27,736 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:27,736 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:27,736 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:27,736 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:27,736 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:27,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on ca92eda5f752c4c8f5e2f01f66bc73d8 2023-04-12 15:54:27,741 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ca92eda5f752c4c8f5e2f01f66bc73d8 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-12 15:54:27,741 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca92eda5f752c4c8f5e2f01f66bc73d8, store=A 2023-04-12 15:54:27,741 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:27,741 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca92eda5f752c4c8f5e2f01f66bc73d8, store=B 2023-04-12 15:54:27,741 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:27,741 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca92eda5f752c4c8f5e2f01f66bc73d8, store=C 2023-04-12 15:54:27,741 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:27,748 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:27,748 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:27,752 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:27,752 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. started... 2023-04-12 15:54:27,752 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. as already flushing 2023-04-12 15:54:27,752 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:27,753 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:27,753 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:27,753 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:27,757 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] throttle.PressureAwareThroughputController(145): ca92eda5f752c4c8f5e2f01f66bc73d8#A#compaction#183 average throughput is 0.48 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:54:27,758 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:27,758 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:27,760 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:27,760 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:27,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:27,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:27,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:27,760 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:27,760 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:27,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:27,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:27,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:27,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:27,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:27,763 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:27,763 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16ad9c73[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:27,763 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:27,763 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:27,763 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/B/de200f27f4f1477b8820b938bb06e6e2 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/de200f27f4f1477b8820b938bb06e6e2 2023-04-12 15:54:27,766 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:27,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:27,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:27,766 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:27,766 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:27,766 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:27,766 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:27,766 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:27,767 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:27,767 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:27,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:27,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:27,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:27,767 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:27,768 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:27,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:27,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:27,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:27,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:27,770 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ca92eda5f752c4c8f5e2f01f66bc73d8/B of ca92eda5f752c4c8f5e2f01f66bc73d8 into de200f27f4f1477b8820b938bb06e6e2(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:54:27,770 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ca92eda5f752c4c8f5e2f01f66bc73d8: 2023-04-12 15:54:27,770 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8., storeName=ca92eda5f752c4c8f5e2f01f66bc73d8/B, priority=13, startTime=1681314867702; duration=0sec 2023-04-12 15:54:27,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:27,770 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:54:27,770 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 15:54:27,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:27,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:27,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:27,771 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:27,771 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:27,771 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:27,771 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:27,772 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:27,772 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:27,772 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37376 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 15:54:27,772 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1912): ca92eda5f752c4c8f5e2f01f66bc73d8/C is initiating minor compaction (all files) 2023-04-12 15:54:27,772 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ca92eda5f752c4c8f5e2f01f66bc73d8/C in TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:27,772 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/af2631c32ea54ed3b42c89ff2e36a03e, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/78a5a72a6f48485b86d486c5ddacb09e, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/c00dea6e6e49410b86fb135e07cebf79] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp, totalSize=36.5 K 2023-04-12 15:54:27,773 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:27,773 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:27,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:27,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:27,773 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:27,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:27,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:27,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:27,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:27,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:27,774 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:27,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:27,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:27,774 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting af2631c32ea54ed3b42c89ff2e36a03e, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=391, earliestPutTs=1681314864616 2023-04-12 15:54:27,775 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:27,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34118 deadline: 1681314927774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:27,776 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 78a5a72a6f48485b86d486c5ddacb09e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=404, earliestPutTs=1681314865748 2023-04-12 15:54:27,776 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting c00dea6e6e49410b86fb135e07cebf79, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=420, earliestPutTs=1681314867561 2023-04-12 15:54:27,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:27,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:27,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:27,777 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:27,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:27,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:27,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:27,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:27,780 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:27,780 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:27,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:27,781 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:27,780 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:27,781 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:27,780 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:27,780 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:27,781 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:27,781 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:27,781 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:27,781 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:27,781 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:27,781 DEBUG [Thread-831] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:27,780 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:27,781 DEBUG [Thread-831] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:27,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:27,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:27,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:27,797 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] throttle.PressureAwareThroughputController(145): ca92eda5f752c4c8f5e2f01f66bc73d8#C#compaction#186 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:54:27,813 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230412ab322cc7ad7c433f98d0ac31ab43e229 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412ab322cc7ad7c433f98d0ac31ab43e229 2023-04-12 15:54:27,844 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=440, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/A/d3c4d3a5ed0d457eb6b1ab4d2a6002fd 2023-04-12 15:54:27,852 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/A/51f413ebb2454f9fb37c2c8d844fcd88 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/51f413ebb2454f9fb37c2c8d844fcd88 2023-04-12 15:54:27,865 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ca92eda5f752c4c8f5e2f01f66bc73d8/A of ca92eda5f752c4c8f5e2f01f66bc73d8 into 51f413ebb2454f9fb37c2c8d844fcd88(size=26.3 K), total size for store is 26.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:54:27,865 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ca92eda5f752c4c8f5e2f01f66bc73d8: 2023-04-12 15:54:27,865 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8., storeName=ca92eda5f752c4c8f5e2f01f66bc73d8/A, priority=13, startTime=1681314867702; duration=0sec 2023-04-12 15:54:27,865 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:54:27,870 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=440 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/B/7eb1ab912a574dfb8827506dd9969a11 2023-04-12 15:54:27,876 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:27,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34094 deadline: 1681314927875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:27,879 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:27,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34118 deadline: 1681314927877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:27,881 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:27,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34134 deadline: 1681314927879, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:27,881 DEBUG [Thread-831] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:27,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:27,882 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:27,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34066 deadline: 1681314927880, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:27,883 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:27,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:27,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:27,886 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:27,886 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:27,886 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:27,887 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:27,887 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:27,887 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:27,893 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:27,894 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:27,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:27,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:27,894 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:27,894 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:27,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:27,894 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:27,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:27,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:27,895 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:27,904 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:27,904 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:27,904 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:27,904 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:27,904 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:27,921 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:27,921 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:27,924 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:27,924 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. started... 2023-04-12 15:54:27,924 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. as already flushing 2023-04-12 15:54:27,924 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:27,924 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:27,924 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:27,924 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:27,924 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:27,924 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:27,928 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:27,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:27,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:27,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:27,928 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:27,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:27,928 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:27,928 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:27,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:27,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:27,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:27,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:27,931 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:27,931 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3550eb0a[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:27,931 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:27,931 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:27,932 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:27,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:27,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:27,933 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:27,933 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:27,933 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:27,933 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:27,933 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:27,938 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:27,938 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:27,938 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:27,938 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:27,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:27,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:27,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:27,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:27,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:27,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:27,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:27,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:27,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:27,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:27,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:27,942 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:27,942 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:27,942 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:27,942 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:27,942 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:27,942 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:27,944 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:27,944 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:27,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:27,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:27,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:27,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:27,944 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:27,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:27,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:27,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:27,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:27,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:27,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:27,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:27,946 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:27,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:27,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:27,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:27,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:27,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:27,949 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:27,952 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:27,952 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:27,952 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:27,952 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:27,952 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:27,952 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:27,952 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:27,952 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:27,952 DEBUG [Thread-831] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:27,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:27,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:27,952 DEBUG [Thread-831] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:27,952 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:27,953 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:27,953 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:27,953 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:27,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:27,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:28,053 DEBUG [Thread-831] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:28,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:28,055 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:28,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:28,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:28,056 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:28,056 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:28,056 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:28,056 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:28,057 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:28,057 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:28,058 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:28,058 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:28,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:28,058 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,059 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:28,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:28,059 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:28,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:28,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:28,059 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:28,060 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:28,060 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:28,060 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:28,060 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:28,060 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:28,060 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:28,060 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:28,060 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:28,060 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. started... 2023-04-12 15:54:28,060 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. as already flushing 2023-04-12 15:54:28,060 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:28,060 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:28,060 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:28,060 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,060 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:28,060 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:28,063 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:28,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:28,063 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:28,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:28,064 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:28,064 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:28,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:28,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:28,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:28,065 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:28,065 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75645894[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:28,065 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:28,065 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:28,066 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:28,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:28,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:28,066 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:28,066 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:28,066 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,067 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:28,066 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:28,068 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:28,068 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:28,068 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,068 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:28,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:28,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:28,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:28,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:28,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:28,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:28,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:28,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:28,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:28,072 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:28,072 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:28,072 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:28,072 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:28,072 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:28,073 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:28,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:28,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:28,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:28,073 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:28,073 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:28,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:28,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:28,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:28,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:28,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:28,074 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:28,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:28,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:28,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:28,075 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:28,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:28,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,079 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,079 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:28,079 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:28,079 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:28,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:28,079 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:28,079 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:28,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:28,079 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:28,079 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:28,079 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:28,079 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,080 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:28,080 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:28,080 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:28,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:28,080 DEBUG [Thread-831] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:28,080 DEBUG [Thread-831] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:28,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:28,081 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:28,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34118 deadline: 1681314928080, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,180 DEBUG [Thread-831] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:28,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:28,182 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:28,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:28,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:28,185 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:28,185 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:28,186 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:28,186 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:28,186 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:28,186 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:28,187 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:28,187 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:28,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:28,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:28,188 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,188 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:28,188 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:28,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:28,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:28,189 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:28,189 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:28,189 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:28,189 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:28,189 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:28,189 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:28,196 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:28,196 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:28,196 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:28,196 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. started... 2023-04-12 15:54:28,196 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. as already flushing 2023-04-12 15:54:28,196 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:28,196 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:28,196 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:28,196 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,196 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:28,196 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:28,198 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:28,198 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:28,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:28,198 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:28,198 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:28,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:28,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:28,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:28,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:28,200 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:28,200 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d03096f[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:28,200 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:28,200 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:28,201 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:28,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:28,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:28,201 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:28,201 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:28,201 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:28,202 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,202 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:28,205 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:28,205 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,205 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:28,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:28,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:28,205 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:28,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:28,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:28,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:28,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:28,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:28,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:28,207 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:28,207 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:28,207 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:28,207 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:28,207 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:28,207 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:28,209 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:28,209 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:28,209 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:28,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:28,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:28,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:28,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:28,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:28,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:28,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:28,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:28,210 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:28,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:28,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:28,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:28,211 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:28,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:28,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,214 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:28,214 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:28,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:28,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:28,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:28,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:28,215 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:28,215 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,215 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:28,215 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:28,215 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:28,215 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:28,215 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,215 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:28,215 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:28,215 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:28,215 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:28,215 DEBUG [Thread-831] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:28,215 DEBUG [Thread-831] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:28,222 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/C/cb3beed26eea442084a31f6e5ff88d44 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/cb3beed26eea442084a31f6e5ff88d44 2023-04-12 15:54:28,229 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ca92eda5f752c4c8f5e2f01f66bc73d8/C of ca92eda5f752c4c8f5e2f01f66bc73d8 into cb3beed26eea442084a31f6e5ff88d44(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:54:28,229 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ca92eda5f752c4c8f5e2f01f66bc73d8: 2023-04-12 15:54:28,229 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8., storeName=ca92eda5f752c4c8f5e2f01f66bc73d8/C, priority=13, startTime=1681314867702; duration=0sec 2023-04-12 15:54:28,229 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:54:28,316 DEBUG [Thread-831] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:28,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:28,318 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:28,318 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=440 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/C/a92fc614903f42ea96291665c91ac723 2023-04-12 15:54:28,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:28,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:28,320 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:28,320 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:28,320 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:28,320 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:28,321 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:28,321 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:28,324 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:28,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:28,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:28,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:28,324 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,325 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:28,325 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,325 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:28,325 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/A/d3c4d3a5ed0d457eb6b1ab4d2a6002fd as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/d3c4d3a5ed0d457eb6b1ab4d2a6002fd 2023-04-12 15:54:28,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:28,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:28,325 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:28,326 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:28,326 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:28,326 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:28,326 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:28,326 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:28,328 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:28,328 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:28,328 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:28,328 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. started... 2023-04-12 15:54:28,328 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. as already flushing 2023-04-12 15:54:28,328 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:28,328 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:28,328 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:28,328 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,328 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:28,328 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:28,331 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:28,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:28,331 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:28,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:28,333 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/d3c4d3a5ed0d457eb6b1ab4d2a6002fd, entries=200, sequenceid=440, filesize=32.3 K 2023-04-12 15:54:28,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:28,333 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:28,334 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:28,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:28,334 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/B/7eb1ab912a574dfb8827506dd9969a11 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/7eb1ab912a574dfb8827506dd9969a11 2023-04-12 15:54:28,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:28,335 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:28,335 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@573756b8[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:28,335 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:28,335 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:28,336 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:28,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:28,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:28,336 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,336 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:28,336 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:28,336 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:28,336 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:28,338 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,338 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:28,338 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:28,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:28,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:28,338 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:28,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:28,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:28,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:28,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:28,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:28,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:28,340 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:28,340 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:28,340 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:28,340 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:28,340 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:28,340 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:28,341 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:28,341 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:28,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:28,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:28,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:28,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:28,341 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:28,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:28,342 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:28,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:28,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:28,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:28,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:28,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:28,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:28,343 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:28,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:28,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,348 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/7eb1ab912a574dfb8827506dd9969a11, entries=150, sequenceid=440, filesize=11.9 K 2023-04-12 15:54:28,349 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/C/a92fc614903f42ea96291665c91ac723 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/a92fc614903f42ea96291665c91ac723 2023-04-12 15:54:28,349 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,350 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:28,350 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:28,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:28,350 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:28,350 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:28,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:28,350 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:28,350 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:28,350 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:28,350 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:28,350 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,350 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:28,350 DEBUG [Thread-831] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:28,350 DEBUG [Thread-831] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:28,350 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:28,350 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:28,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:28,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:28,355 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/a92fc614903f42ea96291665c91ac723, entries=150, sequenceid=440, filesize=11.9 K 2023-04-12 15:54:28,355 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for ca92eda5f752c4c8f5e2f01f66bc73d8 in 614ms, sequenceid=440, compaction requested=false 2023-04-12 15:54:28,356 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ca92eda5f752c4c8f5e2f01f66bc73d8: 2023-04-12 15:54:28,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on ca92eda5f752c4c8f5e2f01f66bc73d8 2023-04-12 15:54:28,391 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ca92eda5f752c4c8f5e2f01f66bc73d8 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-12 15:54:28,391 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca92eda5f752c4c8f5e2f01f66bc73d8, store=A 2023-04-12 15:54:28,391 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:28,391 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca92eda5f752c4c8f5e2f01f66bc73d8, store=B 2023-04-12 15:54:28,391 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:28,391 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca92eda5f752c4c8f5e2f01f66bc73d8, store=C 2023-04-12 15:54:28,391 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:28,450 DEBUG [Thread-831] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:28,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:28,457 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:28,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:28,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:28,459 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:28,459 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:28,459 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:28,459 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:28,459 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:28,459 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:28,460 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:28,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:28,461 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:28,461 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,461 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:28,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:28,462 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:28,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:28,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:28,462 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:28,464 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:28,464 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:28,464 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:28,464 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:28,464 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:28,465 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:28,465 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:28,465 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:28,465 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. started... 2023-04-12 15:54:28,465 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. as already flushing 2023-04-12 15:54:28,465 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:28,465 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:28,465 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:28,465 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,465 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:28,465 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:28,467 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,467 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:28,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:28,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:28,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:28,467 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:28,468 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:28,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:28,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:28,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:28,469 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:28,469 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@650dffef[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:28,469 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:28,469 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:28,470 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:28,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:28,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:28,470 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,470 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:28,470 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:28,471 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:28,471 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:28,474 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,474 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:28,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:28,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:28,474 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:28,474 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:28,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:28,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:28,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:28,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:28,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:28,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:28,476 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:28,476 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:28,476 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:28,476 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:28,476 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:28,476 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:28,478 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:28,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:28,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:28,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:28,478 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:28,478 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:28,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:28,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:28,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:28,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:28,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:28,478 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:28,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:28,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:28,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:28,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,480 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:28,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:28,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,482 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:28,482 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:28,482 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:28,483 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:28,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:28,483 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:28,483 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:28,483 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,483 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:28,483 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:28,483 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:28,483 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:28,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:28,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:28,483 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:28,483 DEBUG [Thread-831] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:28,483 DEBUG [Thread-831] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:28,549 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:28,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34118 deadline: 1681314928548, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,584 DEBUG [Thread-831] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:28,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:28,586 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:28,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:28,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:28,587 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:28,587 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:28,587 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:28,587 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:28,587 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:28,588 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:28,589 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:28,589 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:28,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:28,589 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,589 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:28,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:28,589 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:28,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:28,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:28,590 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:28,590 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:28,590 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:28,590 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:28,590 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:28,590 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:28,590 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:28,590 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:28,591 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:28,591 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. started... 2023-04-12 15:54:28,591 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. as already flushing 2023-04-12 15:54:28,591 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:28,591 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:28,591 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:28,591 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,591 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:28,591 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:28,593 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:28,593 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:28,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:28,593 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:28,594 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:28,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:28,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:28,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:28,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:28,595 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:28,595 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32fabb22[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:28,595 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:28,595 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:28,596 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:28,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:28,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:28,596 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:28,597 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:28,597 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:28,597 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,597 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:28,598 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:28,598 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:28,598 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,598 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:28,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:28,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:28,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:28,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:28,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:28,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:28,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:28,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:28,600 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:28,600 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:28,600 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:28,600 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:28,600 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:28,600 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:28,602 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:28,602 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:28,602 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:28,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:28,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:28,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:28,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:28,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:28,602 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:28,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:28,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:28,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:28,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:28,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:28,603 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:28,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:28,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:28,607 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,607 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:28,607 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:28,607 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:28,607 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:28,607 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:28,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:28,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:28,607 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:28,607 DEBUG [Thread-831] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:28,607 DEBUG [Thread-831] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:28,607 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:28,607 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,607 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:28,607 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:28,607 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:28,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:28,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:28,607 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:28,652 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:28,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34118 deadline: 1681314928651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,707 DEBUG [Thread-831] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:28,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:28,709 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:28,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:28,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:28,711 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:28,711 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:28,711 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:28,711 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:28,711 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:28,711 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:28,712 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,712 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:28,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:28,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:28,713 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,713 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:28,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:28,713 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:28,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:28,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:28,714 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:28,714 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:28,714 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:28,714 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:28,714 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:28,714 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:28,714 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:28,714 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:28,714 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:28,714 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. started... 2023-04-12 15:54:28,715 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. as already flushing 2023-04-12 15:54:28,715 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:28,715 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:28,715 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:28,715 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,715 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:28,715 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:28,716 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,716 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:28,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:28,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:28,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:28,717 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:28,717 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:28,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:28,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:28,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:28,718 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:28,718 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:28,718 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7079463[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:28,718 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:28,719 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:28,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:28,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:28,719 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:28,719 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:28,719 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:28,719 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,719 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:28,722 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,722 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:28,722 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:28,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:28,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:28,722 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:28,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:28,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:28,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:28,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:28,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:28,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:28,724 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:28,724 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:28,724 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:28,724 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:28,724 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:28,724 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:28,726 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:28,726 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:28,726 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:28,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:28,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:28,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:28,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:28,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:28,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:28,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:28,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:28,727 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:28,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:28,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:28,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:28,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,728 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:28,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:28,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,733 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:28,733 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:28,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:28,733 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:28,733 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:28,733 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:28,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:28,733 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:28,733 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,733 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:28,733 DEBUG [Thread-831] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:28,734 DEBUG [Thread-831] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:28,733 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:28,734 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,734 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:28,734 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:28,734 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:28,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:28,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:28,831 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230412ca912ea2732b454caa5aabe7d6946f93 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412ca912ea2732b454caa5aabe7d6946f93 2023-04-12 15:54:28,834 DEBUG [Thread-831] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:28,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:28,837 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:28,837 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=456, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/A/2bca9b34427e44bd895ff91396387ef5 2023-04-12 15:54:28,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:28,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:28,839 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:28,839 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:28,839 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:28,839 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:28,839 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:28,839 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:28,846 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:28,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:28,846 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:28,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:28,847 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,847 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:28,848 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:28,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:28,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:28,848 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:28,848 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:28,848 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:28,848 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:28,849 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:28,849 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:28,849 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:28,849 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:28,849 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:28,849 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. started... 2023-04-12 15:54:28,849 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. as already flushing 2023-04-12 15:54:28,849 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:28,849 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:28,849 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:28,849 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,849 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:28,849 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:28,850 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,851 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:28,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:28,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:28,851 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:28,851 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:28,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:28,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:28,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:28,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,852 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=456 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/B/b47fbbbfd7364685b69495196de64c8d 2023-04-12 15:54:28,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:28,853 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:28,853 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ae880ad[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:28,853 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:28,853 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:28,854 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:28,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:28,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:28,854 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:28,854 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:28,854 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:28,854 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,854 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:28,855 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:28,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34118 deadline: 1681314928855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,856 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:28,856 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:28,856 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:28,857 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:28,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:28,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:28,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:28,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:28,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:28,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:28,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:28,861 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:28,861 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:28,861 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:28,861 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:28,861 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:28,861 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:28,863 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:28,864 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:28,864 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:28,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:28,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:28,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:28,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:28,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:28,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:28,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:28,865 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:28,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:28,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:28,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:28,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:28,866 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:28,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:28,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,869 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,869 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:28,869 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:28,869 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:28,869 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:28,869 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:28,869 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:28,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:28,869 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:28,869 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:28,869 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:28,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:28,869 DEBUG [Thread-831] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:28,869 DEBUG [Thread-831] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:28,869 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:28,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:28,869 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:28,870 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:28,969 DEBUG [Thread-831] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:28,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:28,972 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:28,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:28,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:28,973 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:28,973 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:28,973 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:28,973 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:28,973 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:28,974 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:28,975 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:28,975 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:28,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:28,975 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,975 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:28,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:28,975 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:28,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:28,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:28,976 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:28,976 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:28,976 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:28,976 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:28,976 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:28,976 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:28,977 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:28,977 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:28,977 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:28,977 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. started... 2023-04-12 15:54:28,977 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. as already flushing 2023-04-12 15:54:28,977 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:28,977 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:28,977 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:28,977 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,977 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:28,977 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:28,978 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,978 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:28,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:28,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:28,979 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:28,979 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:28,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:28,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:28,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:28,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:28,980 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:28,980 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6090f898[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:28,980 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:28,981 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:28,982 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:28,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:28,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:28,983 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:28,983 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:28,983 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:28,983 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,983 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:28,984 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,984 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:28,984 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:28,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:28,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:28,984 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:28,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:28,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:28,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:28,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:28,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:28,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:28,986 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:28,986 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:28,987 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:28,987 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:28,987 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:28,987 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:28,988 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:28,988 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:28,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:28,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:28,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:28,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:28,988 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:28,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:28,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:28,988 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:28,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:28,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:28,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:28,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:28,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:28,990 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:28,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:28,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,992 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,992 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:28,992 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:28,992 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:28,992 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:28,992 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:28,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:28,992 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:28,992 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:28,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:28,992 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:28,992 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:28,993 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:28,992 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:28,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:28,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:28,993 DEBUG [Thread-831] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:28,993 DEBUG [Thread-831] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:28,993 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:29,093 DEBUG [Thread-831] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:29,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:29,095 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:29,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:29,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:29,097 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:29,097 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:29,097 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:29,097 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:29,097 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:29,097 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:29,099 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:29,099 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:29,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:29,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:29,100 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:29,100 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:29,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:29,100 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:29,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:29,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:29,101 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:29,101 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:29,101 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:29,101 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:29,101 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:29,101 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:29,102 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:29,102 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:29,102 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:29,102 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. started... 2023-04-12 15:54:29,103 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. as already flushing 2023-04-12 15:54:29,103 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:29,103 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:29,103 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:29,103 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:29,103 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:29,103 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:29,105 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:29,105 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:29,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:29,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:29,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:29,105 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:29,105 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:29,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:29,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:29,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:29,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:29,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:29,106 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:29,106 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d46c03[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:29,106 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:29,106 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:29,107 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:29,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:29,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:29,108 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:29,108 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:29,108 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:29,108 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:29,108 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:29,109 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:29,109 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:29,109 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:29,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:29,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:29,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:29,109 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:29,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:29,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:29,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:29,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:29,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:29,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:29,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:29,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:29,111 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:29,111 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:29,111 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:29,111 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:29,112 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:29,112 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:29,116 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:29,116 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:29,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:29,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:29,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:29,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:29,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:29,116 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:29,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:29,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:29,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:29,117 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:29,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:29,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:29,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:29,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:29,118 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:29,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:29,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:29,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:29,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:29,121 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:29,121 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:29,121 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:29,121 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:29,121 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:29,121 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:29,121 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:29,121 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:29,121 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:29,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:29,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:29,121 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:29,122 DEBUG [Thread-831] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:29,122 DEBUG [Thread-831] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:29,122 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:29,122 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:29,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:29,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:29,123 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:29,161 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:29,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34118 deadline: 1681314929159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:29,170 DEBUG [Thread-836] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x42d325f4 to 127.0.0.1:60154 2023-04-12 15:54:29,170 DEBUG [Thread-836] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 15:54:29,171 DEBUG [Thread-834] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1a7b66a3 to 127.0.0.1:60154 2023-04-12 15:54:29,171 DEBUG [Thread-834] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 15:54:29,172 DEBUG [Thread-832] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4a0fd7b4 to 127.0.0.1:60154 2023-04-12 15:54:29,172 DEBUG [Thread-832] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 15:54:29,173 DEBUG [Thread-838] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1676475f to 127.0.0.1:60154 2023-04-12 15:54:29,173 DEBUG [Thread-838] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 15:54:29,222 DEBUG [Thread-831] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:29,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:29,264 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=456 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/C/9c4b1051659e4964b2e4ea6225353cff 2023-04-12 15:54:29,269 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/A/2bca9b34427e44bd895ff91396387ef5 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/2bca9b34427e44bd895ff91396387ef5 2023-04-12 15:54:29,275 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/2bca9b34427e44bd895ff91396387ef5, entries=150, sequenceid=456, filesize=25.4 K 2023-04-12 15:54:29,275 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/B/b47fbbbfd7364685b69495196de64c8d as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/b47fbbbfd7364685b69495196de64c8d 2023-04-12 15:54:29,279 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/b47fbbbfd7364685b69495196de64c8d, entries=100, sequenceid=456, filesize=9.5 K 2023-04-12 15:54:29,279 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/C/9c4b1051659e4964b2e4ea6225353cff as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/9c4b1051659e4964b2e4ea6225353cff 2023-04-12 15:54:29,283 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/9c4b1051659e4964b2e4ea6225353cff, entries=100, sequenceid=456, filesize=9.5 K 2023-04-12 15:54:29,283 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for ca92eda5f752c4c8f5e2f01f66bc73d8 in 892ms, sequenceid=456, compaction requested=true 2023-04-12 15:54:29,283 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ca92eda5f752c4c8f5e2f01f66bc73d8: 2023-04-12 15:54:29,284 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:54:29,284 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 15:54:29,284 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 15:54:29,284 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:54:29,284 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:54:29,285 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 86094 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 15:54:29,285 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1912): ca92eda5f752c4c8f5e2f01f66bc73d8/A is initiating minor compaction (all files) 2023-04-12 15:54:29,285 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35034 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 15:54:29,285 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ca92eda5f752c4c8f5e2f01f66bc73d8/A in TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:29,285 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1912): ca92eda5f752c4c8f5e2f01f66bc73d8/B is initiating minor compaction (all files) 2023-04-12 15:54:29,285 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/51f413ebb2454f9fb37c2c8d844fcd88, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/d3c4d3a5ed0d457eb6b1ab4d2a6002fd, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/2bca9b34427e44bd895ff91396387ef5] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp, totalSize=84.1 K 2023-04-12 15:54:29,285 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ca92eda5f752c4c8f5e2f01f66bc73d8/B in TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:29,285 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/de200f27f4f1477b8820b938bb06e6e2, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/7eb1ab912a574dfb8827506dd9969a11, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/b47fbbbfd7364685b69495196de64c8d] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp, totalSize=34.2 K 2023-04-12 15:54:29,285 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 51f413ebb2454f9fb37c2c8d844fcd88, keycount=150, bloomtype=ROW, size=26.3 K, encoding=NONE, compression=NONE, seqNum=420, earliestPutTs=1681314867561 2023-04-12 15:54:29,285 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting de200f27f4f1477b8820b938bb06e6e2, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=420, earliestPutTs=1681314867561 2023-04-12 15:54:29,286 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting d3c4d3a5ed0d457eb6b1ab4d2a6002fd, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=440, earliestPutTs=1681314867620 2023-04-12 15:54:29,286 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 7eb1ab912a574dfb8827506dd9969a11, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=440, earliestPutTs=1681314867620 2023-04-12 15:54:29,286 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 2bca9b34427e44bd895ff91396387ef5, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=456, earliestPutTs=1681314867761 2023-04-12 15:54:29,286 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting b47fbbbfd7364685b69495196de64c8d, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=456, earliestPutTs=1681314867761 2023-04-12 15:54:29,294 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] throttle.PressureAwareThroughputController(145): ca92eda5f752c4c8f5e2f01f66bc73d8#B#compaction#193 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:54:29,302 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] throttle.PressureAwareThroughputController(145): ca92eda5f752c4c8f5e2f01f66bc73d8#A#compaction#192 average throughput is 1.97 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:54:29,308 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/B/c4aa19992baa4e9a8b3e40565d71006e as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/c4aa19992baa4e9a8b3e40565d71006e 2023-04-12 15:54:29,311 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ca92eda5f752c4c8f5e2f01f66bc73d8/B of ca92eda5f752c4c8f5e2f01f66bc73d8 into c4aa19992baa4e9a8b3e40565d71006e(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:54:29,312 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ca92eda5f752c4c8f5e2f01f66bc73d8: 2023-04-12 15:54:29,312 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8., storeName=ca92eda5f752c4c8f5e2f01f66bc73d8/B, priority=13, startTime=1681314869284; duration=0sec 2023-04-12 15:54:29,312 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:54:29,312 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 15:54:29,313 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35034 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 15:54:29,313 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1912): ca92eda5f752c4c8f5e2f01f66bc73d8/C is initiating minor compaction (all files) 2023-04-12 15:54:29,313 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ca92eda5f752c4c8f5e2f01f66bc73d8/C in TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:29,313 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/cb3beed26eea442084a31f6e5ff88d44, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/a92fc614903f42ea96291665c91ac723, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/9c4b1051659e4964b2e4ea6225353cff] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp, totalSize=34.2 K 2023-04-12 15:54:29,313 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting cb3beed26eea442084a31f6e5ff88d44, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=420, earliestPutTs=1681314867561 2023-04-12 15:54:29,314 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting a92fc614903f42ea96291665c91ac723, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=440, earliestPutTs=1681314867620 2023-04-12 15:54:29,314 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 9c4b1051659e4964b2e4ea6225353cff, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=456, earliestPutTs=1681314867761 2023-04-12 15:54:29,320 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] throttle.PressureAwareThroughputController(145): ca92eda5f752c4c8f5e2f01f66bc73d8#C#compaction#194 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:54:29,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on ca92eda5f752c4c8f5e2f01f66bc73d8 2023-04-12 15:54:29,665 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ca92eda5f752c4c8f5e2f01f66bc73d8 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-12 15:54:29,665 DEBUG [Thread-829] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x508246cc to 127.0.0.1:60154 2023-04-12 15:54:29,665 DEBUG [Thread-829] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 15:54:29,666 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca92eda5f752c4c8f5e2f01f66bc73d8, store=A 2023-04-12 15:54:29,666 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:29,666 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca92eda5f752c4c8f5e2f01f66bc73d8, store=B 2023-04-12 15:54:29,666 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:29,666 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca92eda5f752c4c8f5e2f01f66bc73d8, store=C 2023-04-12 15:54:29,666 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:29,731 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/C/9a8fd00447604e338cf15680004fcc0a as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/9a8fd00447604e338cf15680004fcc0a 2023-04-12 15:54:29,736 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ca92eda5f752c4c8f5e2f01f66bc73d8/C of ca92eda5f752c4c8f5e2f01f66bc73d8 into 9a8fd00447604e338cf15680004fcc0a(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:54:29,736 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ca92eda5f752c4c8f5e2f01f66bc73d8: 2023-04-12 15:54:29,736 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8., storeName=ca92eda5f752c4c8f5e2f01f66bc73d8/C, priority=13, startTime=1681314869284; duration=0sec 2023-04-12 15:54:29,736 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:54:29,885 DEBUG [Thread-825] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0feb770d to 127.0.0.1:60154 2023-04-12 15:54:29,885 DEBUG [Thread-825] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 15:54:29,889 DEBUG [Thread-827] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x00c2598b to 127.0.0.1:60154 2023-04-12 15:54:29,889 DEBUG [Thread-827] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 15:54:29,897 DEBUG [Thread-823] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7aba8e46 to 127.0.0.1:60154 2023-04-12 15:54:29,898 DEBUG [Thread-823] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 15:54:30,078 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230412e332824be8d74962beb6bf6e513d6fe9 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412e332824be8d74962beb6bf6e513d6fe9 2023-04-12 15:54:30,483 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=477, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/A/690739fa2fbd46e0bb5a3cdd346175d3 2023-04-12 15:54:30,519 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/A/52ed39b007984dd9872006ea6c3ba61b as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/52ed39b007984dd9872006ea6c3ba61b 2023-04-12 15:54:30,523 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ca92eda5f752c4c8f5e2f01f66bc73d8/A of ca92eda5f752c4c8f5e2f01f66bc73d8 into 52ed39b007984dd9872006ea6c3ba61b(size=26.4 K), total size for store is 26.4 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-12 15:54:30,523 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ca92eda5f752c4c8f5e2f01f66bc73d8: 2023-04-12 15:54:30,523 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8., storeName=ca92eda5f752c4c8f5e2f01f66bc73d8/A, priority=13, startTime=1681314869284; duration=1sec 2023-04-12 15:54:30,523 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:54:30,893 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=477 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/B/289cec8f57ea4eca8553307f191d2c04 2023-04-12 15:54:30,903 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=477 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/C/56452a8c44514c42ad4e4118578e11ec 2023-04-12 15:54:30,907 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/A/690739fa2fbd46e0bb5a3cdd346175d3 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/690739fa2fbd46e0bb5a3cdd346175d3 2023-04-12 15:54:30,911 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/690739fa2fbd46e0bb5a3cdd346175d3, entries=150, sequenceid=477, filesize=25.4 K 2023-04-12 15:54:30,911 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/B/289cec8f57ea4eca8553307f191d2c04 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/289cec8f57ea4eca8553307f191d2c04 2023-04-12 15:54:30,915 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/289cec8f57ea4eca8553307f191d2c04, entries=150, sequenceid=477, filesize=11.9 K 2023-04-12 15:54:30,916 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/C/56452a8c44514c42ad4e4118578e11ec as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/56452a8c44514c42ad4e4118578e11ec 2023-04-12 15:54:30,919 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/56452a8c44514c42ad4e4118578e11ec, entries=150, sequenceid=477, filesize=11.9 K 2023-04-12 15:54:30,920 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=20.13 KB/20610 for ca92eda5f752c4c8f5e2f01f66bc73d8 in 1255ms, sequenceid=477, compaction requested=false 2023-04-12 15:54:30,920 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ca92eda5f752c4c8f5e2f01f66bc73d8: 2023-04-12 15:54:35,266 DEBUG [Thread-821] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0a71edf0 to 127.0.0.1:60154 2023-04-12 15:54:35,266 DEBUG [Thread-821] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 15:54:35,267 INFO [Listener at localhost/43583] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-12 15:54:35,267 INFO [Listener at localhost/43583] hbase.AcidGuaranteesTestTool(394): wrote 49 2023-04-12 15:54:35,267 INFO [Listener at localhost/43583] hbase.AcidGuaranteesTestTool(394): wrote 101 2023-04-12 15:54:35,267 INFO [Listener at localhost/43583] hbase.AcidGuaranteesTestTool(394): wrote 90 2023-04-12 15:54:35,267 INFO [Listener at localhost/43583] hbase.AcidGuaranteesTestTool(394): wrote 80 2023-04-12 15:54:35,267 INFO [Listener at localhost/43583] hbase.AcidGuaranteesTestTool(394): wrote 63 2023-04-12 15:54:35,267 INFO [Listener at localhost/43583] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-12 15:54:35,267 INFO [Listener at localhost/43583] hbase.AcidGuaranteesTestTool(398): read 5110 2023-04-12 15:54:35,267 INFO [Listener at localhost/43583] hbase.AcidGuaranteesTestTool(398): read 5109 2023-04-12 15:54:35,267 INFO [Listener at localhost/43583] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-12 15:54:35,267 INFO [Listener at localhost/43583] hbase.AcidGuaranteesTestTool(402): scanned 2173 2023-04-12 15:54:35,267 INFO [Listener at localhost/43583] hbase.AcidGuaranteesTestTool(403): verified 6519 rows 2023-04-12 15:54:35,267 INFO [Listener at localhost/43583] hbase.AcidGuaranteesTestTool(402): scanned 2192 2023-04-12 15:54:35,267 INFO [Listener at localhost/43583] hbase.AcidGuaranteesTestTool(403): verified 6573 rows 2023-04-12 15:54:35,267 INFO [Listener at localhost/43583] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-12 15:54:35,267 DEBUG [Listener at localhost/43583] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5b4fcfe0 to 127.0.0.1:60154 2023-04-12 15:54:35,267 DEBUG [Listener at localhost/43583] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 15:54:35,269 INFO [Listener at localhost/43583] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-12 15:54:35,270 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-12 15:54:35,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure2.ProcedureExecutor(1029): Stored pid=24, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-12 15:54:35,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-12 15:54:35,275 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681314875275"}]},"ts":"1681314875275"} 2023-04-12 15:54:35,276 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-12 15:54:35,278 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-12 15:54:35,279 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=ca92eda5f752c4c8f5e2f01f66bc73d8, UNASSIGN}] 2023-04-12 15:54:35,280 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=ca92eda5f752c4c8f5e2f01f66bc73d8, UNASSIGN 2023-04-12 15:54:35,280 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=ca92eda5f752c4c8f5e2f01f66bc73d8, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:35,281 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681314875280"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681314875280"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681314875280"}]},"ts":"1681314875280"} 2023-04-12 15:54:35,281 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=25, state=RUNNABLE; CloseRegionProcedure ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739}] 2023-04-12 15:54:35,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-12 15:54:35,433 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close ca92eda5f752c4c8f5e2f01f66bc73d8 2023-04-12 15:54:35,433 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing ca92eda5f752c4c8f5e2f01f66bc73d8, disabling compactions & flushes 2023-04-12 15:54:35,433 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:35,433 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:35,433 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. after waiting 0 ms 2023-04-12 15:54:35,433 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:35,433 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing ca92eda5f752c4c8f5e2f01f66bc73d8 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-12 15:54:35,433 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca92eda5f752c4c8f5e2f01f66bc73d8, store=A 2023-04-12 15:54:35,433 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:35,434 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca92eda5f752c4c8f5e2f01f66bc73d8, store=B 2023-04-12 15:54:35,434 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:35,434 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca92eda5f752c4c8f5e2f01f66bc73d8, store=C 2023-04-12 15:54:35,434 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:35,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-12 15:54:35,845 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304127d1fb19f157c4084a90a4b93c3371d5d to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304127d1fb19f157c4084a90a4b93c3371d5d 2023-04-12 15:54:35,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-12 15:54:36,251 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=486, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/A/185c645d6b3b4d7a88965901e96b1dab 2023-04-12 15:54:36,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-12 15:54:36,661 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=486 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/B/6bd4f87942e94c2a96d0e32f706ece01 2023-04-12 15:54:36,670 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=486 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/C/b2f54a8cf71c4712b7b3e588373d136f 2023-04-12 15:54:36,674 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/A/185c645d6b3b4d7a88965901e96b1dab as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/185c645d6b3b4d7a88965901e96b1dab 2023-04-12 15:54:36,677 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/185c645d6b3b4d7a88965901e96b1dab, entries=150, sequenceid=486, filesize=25.4 K 2023-04-12 15:54:36,678 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/B/6bd4f87942e94c2a96d0e32f706ece01 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/6bd4f87942e94c2a96d0e32f706ece01 2023-04-12 15:54:36,682 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/6bd4f87942e94c2a96d0e32f706ece01, entries=150, sequenceid=486, filesize=11.9 K 2023-04-12 15:54:36,682 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/.tmp/C/b2f54a8cf71c4712b7b3e588373d136f as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/b2f54a8cf71c4712b7b3e588373d136f 2023-04-12 15:54:36,686 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/b2f54a8cf71c4712b7b3e588373d136f, entries=150, sequenceid=486, filesize=11.9 K 2023-04-12 15:54:36,687 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for ca92eda5f752c4c8f5e2f01f66bc73d8 in 1254ms, sequenceid=486, compaction requested=true 2023-04-12 15:54:36,708 DEBUG [StoreCloser-TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/241794289536461ab44241ae2b5cd066, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/00a4c9a026ca43578c8fe7ed865f489e, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/8d3fbd974ffd4340b479f0bcf22e0b01, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/5f9d968d7a5c4ce5afc405b82e01abe9, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/5abb434e890b43f8b48a8dfc06182063, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/da539482f9214cc69b85aaee0c00fecc, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/62b43f9b9d574e0882527b21f5e022cc, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/374549cd7be54c8ca75b1c947b75f734, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/94e672eeb69c414ea6c42bd1b8d07eb7, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/7ff490ee87994b168395323b44b276ec, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/5c76f100bfc040a9b61197c36c380c46, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/70f4dcfa2997461b8a9027a10bbc2a90, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/3d54ed87c8084576afaf5a7e004de4ea, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/c6f88585101f45b4a232ba5b2f193cbe, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/036b454ea2034330a02f23811342347b, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/1fe66f000dd448489d9c2e0b2b61f0c2, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/abe818eaef4a4430b42c70a78f5d3cab, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/3d520b66e0a7428bae5f878b06b4f65d, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/ca2d4af540b24fb38a0fa858ddbd6042, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/720dfb88249b4815bd19e329e3c100cb, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/d5c882c855c64cbb8188d8a4d4da2ef9, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/105356904e5f44af9215e2323dca6277, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/f266e1b59bab4f34be489b507bb1145c, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/a0545103cb2c44cda1839f60a7980166, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/da4db90529f14c72bb4c16780e111929, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/74cdd999b19b48978f5ad24bab1c1ff8, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/2b09ca7692ff4b329ed90a266898de5a, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/51f413ebb2454f9fb37c2c8d844fcd88, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/d3c4d3a5ed0d457eb6b1ab4d2a6002fd, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/2bca9b34427e44bd895ff91396387ef5] to archive 2023-04-12 15:54:36,709 DEBUG [StoreCloser-TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-12 15:54:36,711 DEBUG [StoreCloser-TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/241794289536461ab44241ae2b5cd066 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/241794289536461ab44241ae2b5cd066 2023-04-12 15:54:36,712 DEBUG [StoreCloser-TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/00a4c9a026ca43578c8fe7ed865f489e to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/00a4c9a026ca43578c8fe7ed865f489e 2023-04-12 15:54:36,713 DEBUG [StoreCloser-TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/8d3fbd974ffd4340b479f0bcf22e0b01 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/8d3fbd974ffd4340b479f0bcf22e0b01 2023-04-12 15:54:36,714 DEBUG [StoreCloser-TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/5f9d968d7a5c4ce5afc405b82e01abe9 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/5f9d968d7a5c4ce5afc405b82e01abe9 2023-04-12 15:54:36,715 DEBUG [StoreCloser-TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/5abb434e890b43f8b48a8dfc06182063 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/5abb434e890b43f8b48a8dfc06182063 2023-04-12 15:54:36,716 DEBUG [StoreCloser-TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/da539482f9214cc69b85aaee0c00fecc to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/da539482f9214cc69b85aaee0c00fecc 2023-04-12 15:54:36,717 DEBUG [StoreCloser-TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/62b43f9b9d574e0882527b21f5e022cc to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/62b43f9b9d574e0882527b21f5e022cc 2023-04-12 15:54:36,718 DEBUG [StoreCloser-TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/374549cd7be54c8ca75b1c947b75f734 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/374549cd7be54c8ca75b1c947b75f734 2023-04-12 15:54:36,719 DEBUG [StoreCloser-TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/94e672eeb69c414ea6c42bd1b8d07eb7 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/94e672eeb69c414ea6c42bd1b8d07eb7 2023-04-12 15:54:36,720 DEBUG [StoreCloser-TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/7ff490ee87994b168395323b44b276ec to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/7ff490ee87994b168395323b44b276ec 2023-04-12 15:54:36,721 DEBUG [StoreCloser-TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/5c76f100bfc040a9b61197c36c380c46 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/5c76f100bfc040a9b61197c36c380c46 2023-04-12 15:54:36,721 DEBUG [StoreCloser-TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/70f4dcfa2997461b8a9027a10bbc2a90 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/70f4dcfa2997461b8a9027a10bbc2a90 2023-04-12 15:54:36,722 DEBUG [StoreCloser-TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/3d54ed87c8084576afaf5a7e004de4ea to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/3d54ed87c8084576afaf5a7e004de4ea 2023-04-12 15:54:36,723 DEBUG [StoreCloser-TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/c6f88585101f45b4a232ba5b2f193cbe to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/c6f88585101f45b4a232ba5b2f193cbe 2023-04-12 15:54:36,724 DEBUG [StoreCloser-TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/036b454ea2034330a02f23811342347b to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/036b454ea2034330a02f23811342347b 2023-04-12 15:54:36,726 DEBUG [StoreCloser-TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/1fe66f000dd448489d9c2e0b2b61f0c2 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/1fe66f000dd448489d9c2e0b2b61f0c2 2023-04-12 15:54:36,727 DEBUG [StoreCloser-TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/abe818eaef4a4430b42c70a78f5d3cab to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/abe818eaef4a4430b42c70a78f5d3cab 2023-04-12 15:54:36,728 DEBUG [StoreCloser-TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/3d520b66e0a7428bae5f878b06b4f65d to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/3d520b66e0a7428bae5f878b06b4f65d 2023-04-12 15:54:36,729 DEBUG [StoreCloser-TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/ca2d4af540b24fb38a0fa858ddbd6042 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/ca2d4af540b24fb38a0fa858ddbd6042 2023-04-12 15:54:36,730 DEBUG [StoreCloser-TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/720dfb88249b4815bd19e329e3c100cb to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/720dfb88249b4815bd19e329e3c100cb 2023-04-12 15:54:36,731 DEBUG [StoreCloser-TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/d5c882c855c64cbb8188d8a4d4da2ef9 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/d5c882c855c64cbb8188d8a4d4da2ef9 2023-04-12 15:54:36,732 DEBUG [StoreCloser-TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/105356904e5f44af9215e2323dca6277 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/105356904e5f44af9215e2323dca6277 2023-04-12 15:54:36,733 DEBUG [StoreCloser-TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/f266e1b59bab4f34be489b507bb1145c to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/f266e1b59bab4f34be489b507bb1145c 2023-04-12 15:54:36,734 DEBUG [StoreCloser-TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/a0545103cb2c44cda1839f60a7980166 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/a0545103cb2c44cda1839f60a7980166 2023-04-12 15:54:36,735 DEBUG [StoreCloser-TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/da4db90529f14c72bb4c16780e111929 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/da4db90529f14c72bb4c16780e111929 2023-04-12 15:54:36,736 DEBUG [StoreCloser-TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/74cdd999b19b48978f5ad24bab1c1ff8 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/74cdd999b19b48978f5ad24bab1c1ff8 2023-04-12 15:54:36,737 DEBUG [StoreCloser-TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/2b09ca7692ff4b329ed90a266898de5a to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/2b09ca7692ff4b329ed90a266898de5a 2023-04-12 15:54:36,738 DEBUG [StoreCloser-TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/51f413ebb2454f9fb37c2c8d844fcd88 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/51f413ebb2454f9fb37c2c8d844fcd88 2023-04-12 15:54:36,739 DEBUG [StoreCloser-TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/d3c4d3a5ed0d457eb6b1ab4d2a6002fd to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/d3c4d3a5ed0d457eb6b1ab4d2a6002fd 2023-04-12 15:54:36,740 DEBUG [StoreCloser-TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/2bca9b34427e44bd895ff91396387ef5 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/2bca9b34427e44bd895ff91396387ef5 2023-04-12 15:54:36,743 DEBUG [StoreCloser-TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/890822e2f7634ea9b23c646406f620ce, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/22cdd7ba8a3c4e3b8a139e86d83e2f2b, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/b6eb09dc371b4a5fa98c56978e88aaa0, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/46851ff41db6423aaa9e6579f679b062, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/9f1c46d5ce854a2b88438c74d405d1e7, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/841a0cf8efa64001929f1be245bbdbb0, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/4bae1d6cd92f4824844d04d8edbd5aa6, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/65348727ec61498aa81911cb51be9bd1, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/83437973030343d19c239e466a565a4c, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/5691f3347f7b4b9190772503c31e3a70, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/d1413f20510a46dc9d1d84e13ecb0083, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/22856aac047f40a280698be347f4fe95, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/9bf16651b56c467eb714602de7d72c09, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/f5960bdffe644342a31f54fbbb56e08c, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/3f83afdc81fe4a78b07feb0d46df5e59, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/9e71c39221624c21aaf45b199245c1e7, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/380fa9ee9be9437a9299d780b4cc88b8, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/8b50d0c3dcdb423d92caa6e6db2b61c9, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/6da84a2049ff425a82b8a58b489dba73, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/5ac59dfdb9d44efd8f790ec7d0631ca1, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/0de51cfd163947f983b88582735cdeb1, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/f76cbe9988cf4572a5655f8b17bad887, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/65b43acc607c4ef2b4e287077707fce9, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/e63b50ac824b437f9e2f8e02d82948d7, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/8e672c6e6ce34a2592a808228e2504de, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/f149ccb0b7d44a7fad81f2d31dc61c73, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/de200f27f4f1477b8820b938bb06e6e2, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/3668822b22f6428c8143d722c7504c01, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/7eb1ab912a574dfb8827506dd9969a11, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/b47fbbbfd7364685b69495196de64c8d] to archive 2023-04-12 15:54:36,744 DEBUG [StoreCloser-TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-12 15:54:36,746 DEBUG [StoreCloser-TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/890822e2f7634ea9b23c646406f620ce to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/890822e2f7634ea9b23c646406f620ce 2023-04-12 15:54:36,747 DEBUG [StoreCloser-TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/22cdd7ba8a3c4e3b8a139e86d83e2f2b to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/22cdd7ba8a3c4e3b8a139e86d83e2f2b 2023-04-12 15:54:36,748 DEBUG [StoreCloser-TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/b6eb09dc371b4a5fa98c56978e88aaa0 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/b6eb09dc371b4a5fa98c56978e88aaa0 2023-04-12 15:54:36,749 DEBUG [StoreCloser-TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/46851ff41db6423aaa9e6579f679b062 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/46851ff41db6423aaa9e6579f679b062 2023-04-12 15:54:36,750 DEBUG [StoreCloser-TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/9f1c46d5ce854a2b88438c74d405d1e7 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/9f1c46d5ce854a2b88438c74d405d1e7 2023-04-12 15:54:36,751 DEBUG [StoreCloser-TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/841a0cf8efa64001929f1be245bbdbb0 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/841a0cf8efa64001929f1be245bbdbb0 2023-04-12 15:54:36,752 DEBUG [StoreCloser-TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/4bae1d6cd92f4824844d04d8edbd5aa6 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/4bae1d6cd92f4824844d04d8edbd5aa6 2023-04-12 15:54:36,753 DEBUG [StoreCloser-TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/65348727ec61498aa81911cb51be9bd1 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/65348727ec61498aa81911cb51be9bd1 2023-04-12 15:54:36,754 DEBUG [StoreCloser-TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/83437973030343d19c239e466a565a4c to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/83437973030343d19c239e466a565a4c 2023-04-12 15:54:36,754 DEBUG [StoreCloser-TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/5691f3347f7b4b9190772503c31e3a70 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/5691f3347f7b4b9190772503c31e3a70 2023-04-12 15:54:36,755 DEBUG [StoreCloser-TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/d1413f20510a46dc9d1d84e13ecb0083 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/d1413f20510a46dc9d1d84e13ecb0083 2023-04-12 15:54:36,757 DEBUG [StoreCloser-TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/22856aac047f40a280698be347f4fe95 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/22856aac047f40a280698be347f4fe95 2023-04-12 15:54:36,758 DEBUG [StoreCloser-TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/9bf16651b56c467eb714602de7d72c09 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/9bf16651b56c467eb714602de7d72c09 2023-04-12 15:54:36,759 DEBUG [StoreCloser-TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/f5960bdffe644342a31f54fbbb56e08c to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/f5960bdffe644342a31f54fbbb56e08c 2023-04-12 15:54:36,760 DEBUG [StoreCloser-TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/3f83afdc81fe4a78b07feb0d46df5e59 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/3f83afdc81fe4a78b07feb0d46df5e59 2023-04-12 15:54:36,761 DEBUG [StoreCloser-TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/9e71c39221624c21aaf45b199245c1e7 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/9e71c39221624c21aaf45b199245c1e7 2023-04-12 15:54:36,762 DEBUG [StoreCloser-TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/380fa9ee9be9437a9299d780b4cc88b8 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/380fa9ee9be9437a9299d780b4cc88b8 2023-04-12 15:54:36,764 DEBUG [StoreCloser-TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/8b50d0c3dcdb423d92caa6e6db2b61c9 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/8b50d0c3dcdb423d92caa6e6db2b61c9 2023-04-12 15:54:36,765 DEBUG [StoreCloser-TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/6da84a2049ff425a82b8a58b489dba73 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/6da84a2049ff425a82b8a58b489dba73 2023-04-12 15:54:36,766 DEBUG [StoreCloser-TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/5ac59dfdb9d44efd8f790ec7d0631ca1 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/5ac59dfdb9d44efd8f790ec7d0631ca1 2023-04-12 15:54:36,767 DEBUG [StoreCloser-TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/0de51cfd163947f983b88582735cdeb1 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/0de51cfd163947f983b88582735cdeb1 2023-04-12 15:54:36,768 DEBUG [StoreCloser-TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/f76cbe9988cf4572a5655f8b17bad887 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/f76cbe9988cf4572a5655f8b17bad887 2023-04-12 15:54:36,769 DEBUG [StoreCloser-TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/65b43acc607c4ef2b4e287077707fce9 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/65b43acc607c4ef2b4e287077707fce9 2023-04-12 15:54:36,770 DEBUG [StoreCloser-TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/e63b50ac824b437f9e2f8e02d82948d7 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/e63b50ac824b437f9e2f8e02d82948d7 2023-04-12 15:54:36,772 DEBUG [StoreCloser-TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/8e672c6e6ce34a2592a808228e2504de to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/8e672c6e6ce34a2592a808228e2504de 2023-04-12 15:54:36,773 DEBUG [StoreCloser-TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/f149ccb0b7d44a7fad81f2d31dc61c73 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/f149ccb0b7d44a7fad81f2d31dc61c73 2023-04-12 15:54:36,774 DEBUG [StoreCloser-TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/de200f27f4f1477b8820b938bb06e6e2 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/de200f27f4f1477b8820b938bb06e6e2 2023-04-12 15:54:36,775 DEBUG [StoreCloser-TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/3668822b22f6428c8143d722c7504c01 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/3668822b22f6428c8143d722c7504c01 2023-04-12 15:54:36,776 DEBUG [StoreCloser-TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/7eb1ab912a574dfb8827506dd9969a11 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/7eb1ab912a574dfb8827506dd9969a11 2023-04-12 15:54:36,777 DEBUG [StoreCloser-TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/b47fbbbfd7364685b69495196de64c8d to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/b47fbbbfd7364685b69495196de64c8d 2023-04-12 15:54:36,780 DEBUG [StoreCloser-TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/85933e793f944cde9c58b7cfdd679f30, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/da84f2ab521b4d89bedc4299dd71deba, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/e0f9f5ef76834f83a88df39bfdf88372, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/110ced44107f4298a942f0f97c4a9666, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/4ed8ee44d96b4498869b3612f17a7985, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/93baa4de2d5042c2a2bd03d3ea36df9c, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/5611fc74272649b39d04a33911779ca3, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/71bc051ce19a4e018b6c2e31c6d28c04, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/eaaeb56b20b14f5fb63fc8680ac76872, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/e46783633842420aba5c9edfe4553aca, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/c6bef8e944eb405aa018c1fee08f16d6, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/cdb9d721dd7246fe98d1f16a35f98b4f, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/9de195298436425e9801f8848b24ec6e, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/ed513faf841744dd895bbf0cbbfec864, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/128e5a99e00b4744a8632b8a68c5d339, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/cb299058d1de45b7b5158c8881775503, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/a31d3f51858349f09710fcf38bd09889, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/c2f49a8f5d6d44558e087af982d82ad8, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/37acf76c211043079819a4bb57ae55e7, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/49ff5b64306148729f8bdbe9ddc846f9, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/2e626e11a9c44828b4bdac944edba0d4, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/8ff56a57419448128f72f8e54a9be6ef, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/3b8af9470c3644ce86ff6df3155798c1, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/af2631c32ea54ed3b42c89ff2e36a03e, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/6e75b143a8f344beab12241bcb11ee71, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/78a5a72a6f48485b86d486c5ddacb09e, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/cb3beed26eea442084a31f6e5ff88d44, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/c00dea6e6e49410b86fb135e07cebf79, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/a92fc614903f42ea96291665c91ac723, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/9c4b1051659e4964b2e4ea6225353cff] to archive 2023-04-12 15:54:36,781 DEBUG [StoreCloser-TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-12 15:54:36,783 DEBUG [StoreCloser-TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/85933e793f944cde9c58b7cfdd679f30 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/85933e793f944cde9c58b7cfdd679f30 2023-04-12 15:54:36,784 DEBUG [StoreCloser-TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/da84f2ab521b4d89bedc4299dd71deba to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/da84f2ab521b4d89bedc4299dd71deba 2023-04-12 15:54:36,785 DEBUG [StoreCloser-TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/e0f9f5ef76834f83a88df39bfdf88372 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/e0f9f5ef76834f83a88df39bfdf88372 2023-04-12 15:54:36,786 DEBUG [StoreCloser-TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/110ced44107f4298a942f0f97c4a9666 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/110ced44107f4298a942f0f97c4a9666 2023-04-12 15:54:36,788 DEBUG [StoreCloser-TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/4ed8ee44d96b4498869b3612f17a7985 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/4ed8ee44d96b4498869b3612f17a7985 2023-04-12 15:54:36,789 DEBUG [StoreCloser-TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/93baa4de2d5042c2a2bd03d3ea36df9c to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/93baa4de2d5042c2a2bd03d3ea36df9c 2023-04-12 15:54:36,790 DEBUG [StoreCloser-TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/5611fc74272649b39d04a33911779ca3 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/5611fc74272649b39d04a33911779ca3 2023-04-12 15:54:36,791 DEBUG [StoreCloser-TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/71bc051ce19a4e018b6c2e31c6d28c04 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/71bc051ce19a4e018b6c2e31c6d28c04 2023-04-12 15:54:36,793 DEBUG [StoreCloser-TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/eaaeb56b20b14f5fb63fc8680ac76872 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/eaaeb56b20b14f5fb63fc8680ac76872 2023-04-12 15:54:36,794 DEBUG [StoreCloser-TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/e46783633842420aba5c9edfe4553aca to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/e46783633842420aba5c9edfe4553aca 2023-04-12 15:54:36,795 DEBUG [StoreCloser-TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/c6bef8e944eb405aa018c1fee08f16d6 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/c6bef8e944eb405aa018c1fee08f16d6 2023-04-12 15:54:36,796 DEBUG [StoreCloser-TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/cdb9d721dd7246fe98d1f16a35f98b4f to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/cdb9d721dd7246fe98d1f16a35f98b4f 2023-04-12 15:54:36,797 DEBUG [StoreCloser-TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/9de195298436425e9801f8848b24ec6e to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/9de195298436425e9801f8848b24ec6e 2023-04-12 15:54:36,798 DEBUG [StoreCloser-TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/ed513faf841744dd895bbf0cbbfec864 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/ed513faf841744dd895bbf0cbbfec864 2023-04-12 15:54:36,800 DEBUG [StoreCloser-TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/128e5a99e00b4744a8632b8a68c5d339 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/128e5a99e00b4744a8632b8a68c5d339 2023-04-12 15:54:36,801 DEBUG [StoreCloser-TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/cb299058d1de45b7b5158c8881775503 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/cb299058d1de45b7b5158c8881775503 2023-04-12 15:54:36,803 DEBUG [StoreCloser-TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/a31d3f51858349f09710fcf38bd09889 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/a31d3f51858349f09710fcf38bd09889 2023-04-12 15:54:36,804 DEBUG [StoreCloser-TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/c2f49a8f5d6d44558e087af982d82ad8 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/c2f49a8f5d6d44558e087af982d82ad8 2023-04-12 15:54:36,806 DEBUG [StoreCloser-TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/37acf76c211043079819a4bb57ae55e7 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/37acf76c211043079819a4bb57ae55e7 2023-04-12 15:54:36,807 DEBUG [StoreCloser-TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/49ff5b64306148729f8bdbe9ddc846f9 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/49ff5b64306148729f8bdbe9ddc846f9 2023-04-12 15:54:36,807 DEBUG [StoreCloser-TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/2e626e11a9c44828b4bdac944edba0d4 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/2e626e11a9c44828b4bdac944edba0d4 2023-04-12 15:54:36,808 DEBUG [StoreCloser-TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/8ff56a57419448128f72f8e54a9be6ef to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/8ff56a57419448128f72f8e54a9be6ef 2023-04-12 15:54:36,809 DEBUG [StoreCloser-TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/3b8af9470c3644ce86ff6df3155798c1 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/3b8af9470c3644ce86ff6df3155798c1 2023-04-12 15:54:36,810 DEBUG [StoreCloser-TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/af2631c32ea54ed3b42c89ff2e36a03e to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/af2631c32ea54ed3b42c89ff2e36a03e 2023-04-12 15:54:36,811 DEBUG [StoreCloser-TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/6e75b143a8f344beab12241bcb11ee71 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/6e75b143a8f344beab12241bcb11ee71 2023-04-12 15:54:36,812 DEBUG [StoreCloser-TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/78a5a72a6f48485b86d486c5ddacb09e to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/78a5a72a6f48485b86d486c5ddacb09e 2023-04-12 15:54:36,813 DEBUG [StoreCloser-TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/cb3beed26eea442084a31f6e5ff88d44 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/cb3beed26eea442084a31f6e5ff88d44 2023-04-12 15:54:36,814 DEBUG [StoreCloser-TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/c00dea6e6e49410b86fb135e07cebf79 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/c00dea6e6e49410b86fb135e07cebf79 2023-04-12 15:54:36,815 DEBUG [StoreCloser-TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/a92fc614903f42ea96291665c91ac723 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/a92fc614903f42ea96291665c91ac723 2023-04-12 15:54:36,816 DEBUG [StoreCloser-TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/9c4b1051659e4964b2e4ea6225353cff to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/9c4b1051659e4964b2e4ea6225353cff 2023-04-12 15:54:36,823 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/recovered.edits/489.seqid, newMaxSeqId=489, maxSeqId=4 2023-04-12 15:54:36,824 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8. 2023-04-12 15:54:36,824 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for ca92eda5f752c4c8f5e2f01f66bc73d8: 2023-04-12 15:54:36,826 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed ca92eda5f752c4c8f5e2f01f66bc73d8 2023-04-12 15:54:36,826 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=ca92eda5f752c4c8f5e2f01f66bc73d8, regionState=CLOSED 2023-04-12 15:54:36,826 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681314876826"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681314876826"}]},"ts":"1681314876826"} 2023-04-12 15:54:36,828 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=25 2023-04-12 15:54:36,828 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=25, state=SUCCESS; CloseRegionProcedure ca92eda5f752c4c8f5e2f01f66bc73d8, server=jenkins-hbase4.apache.org,38905,1681314818739 in 1.5460 sec 2023-04-12 15:54:36,830 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=24 2023-04-12 15:54:36,830 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=24, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=ca92eda5f752c4c8f5e2f01f66bc73d8, UNASSIGN in 1.5490 sec 2023-04-12 15:54:36,830 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681314876830"}]},"ts":"1681314876830"} 2023-04-12 15:54:36,831 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-12 15:54:36,833 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-12 15:54:36,834 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=24, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.5640 sec 2023-04-12 15:54:37,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-12 15:54:37,379 INFO [Listener at localhost/43583] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 24 completed 2023-04-12 15:54:37,379 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-12 15:54:37,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure2.ProcedureExecutor(1029): Stored pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-12 15:54:37,382 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-12 15:54:37,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-12 15:54:37,382 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=27, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-12 15:54:37,387 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8 2023-04-12 15:54:37,389 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A, FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B, FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C, FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/recovered.edits] 2023-04-12 15:54:37,391 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/185c645d6b3b4d7a88965901e96b1dab to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/185c645d6b3b4d7a88965901e96b1dab 2023-04-12 15:54:37,392 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/52ed39b007984dd9872006ea6c3ba61b to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/52ed39b007984dd9872006ea6c3ba61b 2023-04-12 15:54:37,393 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/690739fa2fbd46e0bb5a3cdd346175d3 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/A/690739fa2fbd46e0bb5a3cdd346175d3 2023-04-12 15:54:37,395 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/289cec8f57ea4eca8553307f191d2c04 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/289cec8f57ea4eca8553307f191d2c04 2023-04-12 15:54:37,396 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/6bd4f87942e94c2a96d0e32f706ece01 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/6bd4f87942e94c2a96d0e32f706ece01 2023-04-12 15:54:37,397 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/c4aa19992baa4e9a8b3e40565d71006e to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/B/c4aa19992baa4e9a8b3e40565d71006e 2023-04-12 15:54:37,399 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/56452a8c44514c42ad4e4118578e11ec to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/56452a8c44514c42ad4e4118578e11ec 2023-04-12 15:54:37,400 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/9a8fd00447604e338cf15680004fcc0a to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/9a8fd00447604e338cf15680004fcc0a 2023-04-12 15:54:37,401 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/b2f54a8cf71c4712b7b3e588373d136f to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/C/b2f54a8cf71c4712b7b3e588373d136f 2023-04-12 15:54:37,403 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/recovered.edits/489.seqid to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8/recovered.edits/489.seqid 2023-04-12 15:54:37,403 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/ca92eda5f752c4c8f5e2f01f66bc73d8 2023-04-12 15:54:37,403 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-12 15:54:37,403 DEBUG [PEWorker-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-12 15:54:37,404 DEBUG [PEWorker-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-12 15:54:37,407 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041207a3a1f6ae4b40d5895c2c553e6b1a68 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041207a3a1f6ae4b40d5895c2c553e6b1a68 2023-04-12 15:54:37,408 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304120926341ac65042f7acd1f7577cdba08c to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304120926341ac65042f7acd1f7577cdba08c 2023-04-12 15:54:37,409 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304121040e86e256e44e681d6f44283e81506 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304121040e86e256e44e681d6f44283e81506 2023-04-12 15:54:37,410 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304121bee7472e58b4912a993c578954f0739 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304121bee7472e58b4912a993c578954f0739 2023-04-12 15:54:37,411 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412366f143f618248b0997f9b783d4b800e to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412366f143f618248b0997f9b783d4b800e 2023-04-12 15:54:37,412 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412414669363df940c587bfc1d662b24e5e to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412414669363df940c587bfc1d662b24e5e 2023-04-12 15:54:37,413 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041257d1a2a7b24b4e3ab01fda16c035537e to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041257d1a2a7b24b4e3ab01fda16c035537e 2023-04-12 15:54:37,414 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304127abb732ee36f4b4ba914030bed87d15f to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304127abb732ee36f4b4ba914030bed87d15f 2023-04-12 15:54:37,415 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304127d1fb19f157c4084a90a4b93c3371d5d to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304127d1fb19f157c4084a90a4b93c3371d5d 2023-04-12 15:54:37,416 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304129963f0c94f084974b9f55178f3a1cc88 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304129963f0c94f084974b9f55178f3a1cc88 2023-04-12 15:54:37,417 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304129a3aeaf36ced4b66af71b5813c9fd285 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304129a3aeaf36ced4b66af71b5813c9fd285 2023-04-12 15:54:37,418 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412a64c1b3e1ff94faeb25fed29c173652c to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412a64c1b3e1ff94faeb25fed29c173652c 2023-04-12 15:54:37,419 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412a72d8d2cb4814a39a29747baf3ae860e to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412a72d8d2cb4814a39a29747baf3ae860e 2023-04-12 15:54:37,420 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412ab322cc7ad7c433f98d0ac31ab43e229 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412ab322cc7ad7c433f98d0ac31ab43e229 2023-04-12 15:54:37,421 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412adfe26944f2f4de2bc7e1c70d54f5641 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412adfe26944f2f4de2bc7e1c70d54f5641 2023-04-12 15:54:37,422 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412b5478d3adf45440dba4f53e6c1756db4 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412b5478d3adf45440dba4f53e6c1756db4 2023-04-12 15:54:37,423 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412bda3e6d797af4e30b2f11f91932e1596 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412bda3e6d797af4e30b2f11f91932e1596 2023-04-12 15:54:37,424 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412c15e05e4381a4cb99d7331f030b351ea to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412c15e05e4381a4cb99d7331f030b351ea 2023-04-12 15:54:37,425 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412ca912ea2732b454caa5aabe7d6946f93 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412ca912ea2732b454caa5aabe7d6946f93 2023-04-12 15:54:37,426 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412d018f45a34324780bc590cd43e64f3a4 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412d018f45a34324780bc590cd43e64f3a4 2023-04-12 15:54:37,427 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412d5e95cc3ecf940cfb2ea1c1c754691a2 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412d5e95cc3ecf940cfb2ea1c1c754691a2 2023-04-12 15:54:37,428 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412d6ab4d6f7bbd4ae8b726d5f9ec89d834 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412d6ab4d6f7bbd4ae8b726d5f9ec89d834 2023-04-12 15:54:37,429 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412e332824be8d74962beb6bf6e513d6fe9 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412e332824be8d74962beb6bf6e513d6fe9 2023-04-12 15:54:37,430 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412e7c60fa217ef46b1927588fcf3c28de2 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412e7c60fa217ef46b1927588fcf3c28de2 2023-04-12 15:54:37,432 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412f8614ba0066c4a13a279be61be8c6dfc to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412f8614ba0066c4a13a279be61be8c6dfc 2023-04-12 15:54:37,432 DEBUG [PEWorker-1] backup.HFileArchiver(596): Deleted hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-12 15:54:37,434 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=27, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-12 15:54:37,435 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-12 15:54:37,437 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-12 15:54:37,438 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=27, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-12 15:54:37,438 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-12 15:54:37,438 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681314877438"}]},"ts":"9223372036854775807"} 2023-04-12 15:54:37,440 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-12 15:54:37,440 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => ca92eda5f752c4c8f5e2f01f66bc73d8, NAME => 'TestAcidGuarantees,,1681314847423.ca92eda5f752c4c8f5e2f01f66bc73d8.', STARTKEY => '', ENDKEY => ''}] 2023-04-12 15:54:37,440 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-12 15:54:37,440 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681314877440"}]},"ts":"9223372036854775807"} 2023-04-12 15:54:37,445 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-12 15:54:37,446 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=27, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-12 15:54:37,447 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=27, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 68 msec 2023-04-12 15:54:37,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-12 15:54:37,483 INFO [Listener at localhost/43583] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 27 completed 2023-04-12 15:54:37,495 INFO [Listener at localhost/43583] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=231 (was 238), OpenFileDescriptor=570 (was 578), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=437 (was 459), ProcessCount=169 (was 169), AvailableMemoryMB=558 (was 671) 2023-04-12 15:54:37,506 INFO [Listener at localhost/43583] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=231, OpenFileDescriptor=570, MaxFileDescriptor=60000, SystemLoadAverage=437, ProcessCount=169, AvailableMemoryMB=558 2023-04-12 15:54:37,508 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-12 15:54:37,508 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-12 15:54:37,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure2.ProcedureExecutor(1029): Stored pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-12 15:54:37,510 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-12 15:54:37,511 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 28 2023-04-12 15:54:37,511 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-12 15:54:37,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-12 15:54:37,512 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4 2023-04-12 15:54:37,513 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4 empty. 2023-04-12 15:54:37,513 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4 2023-04-12 15:54:37,513 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-12 15:54:37,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-12 15:54:37,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-12 15:54:37,921 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-12 15:54:37,923 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => deda1ce51fef7ef8d26da255318c55e4, NAME => 'TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp 2023-04-12 15:54:38,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-12 15:54:38,329 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-12 15:54:38,329 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing deda1ce51fef7ef8d26da255318c55e4, disabling compactions & flushes 2023-04-12 15:54:38,329 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:38,329 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:38,329 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. after waiting 0 ms 2023-04-12 15:54:38,329 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:38,329 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:38,329 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for deda1ce51fef7ef8d26da255318c55e4: 2023-04-12 15:54:38,331 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-12 15:54:38,332 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681314878332"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681314878332"}]},"ts":"1681314878332"} 2023-04-12 15:54:38,333 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-12 15:54:38,334 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-12 15:54:38,334 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681314878334"}]},"ts":"1681314878334"} 2023-04-12 15:54:38,335 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-12 15:54:38,338 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=deda1ce51fef7ef8d26da255318c55e4, ASSIGN}] 2023-04-12 15:54:38,339 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=deda1ce51fef7ef8d26da255318c55e4, ASSIGN 2023-04-12 15:54:38,340 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=deda1ce51fef7ef8d26da255318c55e4, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,38905,1681314818739; forceNewPlan=false, retain=false 2023-04-12 15:54:38,491 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=deda1ce51fef7ef8d26da255318c55e4, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:38,491 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681314878491"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681314878491"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681314878491"}]},"ts":"1681314878491"} 2023-04-12 15:54:38,492 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=29, state=RUNNABLE; OpenRegionProcedure deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739}] 2023-04-12 15:54:38,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-12 15:54:38,647 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:38,647 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => deda1ce51fef7ef8d26da255318c55e4, NAME => 'TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.', STARTKEY => '', ENDKEY => ''} 2023-04-12 15:54:38,648 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees deda1ce51fef7ef8d26da255318c55e4 2023-04-12 15:54:38,648 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-12 15:54:38,648 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for deda1ce51fef7ef8d26da255318c55e4 2023-04-12 15:54:38,648 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for deda1ce51fef7ef8d26da255318c55e4 2023-04-12 15:54:38,649 INFO [StoreOpener-deda1ce51fef7ef8d26da255318c55e4-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region deda1ce51fef7ef8d26da255318c55e4 2023-04-12 15:54:38,650 DEBUG [StoreOpener-deda1ce51fef7ef8d26da255318c55e4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A 2023-04-12 15:54:38,650 DEBUG [StoreOpener-deda1ce51fef7ef8d26da255318c55e4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A 2023-04-12 15:54:38,651 INFO [StoreOpener-deda1ce51fef7ef8d26da255318c55e4-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-12 15:54:38,651 INFO [StoreOpener-deda1ce51fef7ef8d26da255318c55e4-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region deda1ce51fef7ef8d26da255318c55e4 columnFamilyName A 2023-04-12 15:54:38,651 INFO [StoreOpener-deda1ce51fef7ef8d26da255318c55e4-1] regionserver.HStore(310): Store=deda1ce51fef7ef8d26da255318c55e4/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 15:54:38,651 INFO [StoreOpener-deda1ce51fef7ef8d26da255318c55e4-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region deda1ce51fef7ef8d26da255318c55e4 2023-04-12 15:54:38,652 DEBUG [StoreOpener-deda1ce51fef7ef8d26da255318c55e4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B 2023-04-12 15:54:38,653 DEBUG [StoreOpener-deda1ce51fef7ef8d26da255318c55e4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B 2023-04-12 15:54:38,653 INFO [StoreOpener-deda1ce51fef7ef8d26da255318c55e4-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-12 15:54:38,653 INFO [StoreOpener-deda1ce51fef7ef8d26da255318c55e4-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region deda1ce51fef7ef8d26da255318c55e4 columnFamilyName B 2023-04-12 15:54:38,653 INFO [StoreOpener-deda1ce51fef7ef8d26da255318c55e4-1] regionserver.HStore(310): Store=deda1ce51fef7ef8d26da255318c55e4/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 15:54:38,653 INFO [StoreOpener-deda1ce51fef7ef8d26da255318c55e4-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region deda1ce51fef7ef8d26da255318c55e4 2023-04-12 15:54:38,654 DEBUG [StoreOpener-deda1ce51fef7ef8d26da255318c55e4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C 2023-04-12 15:54:38,654 DEBUG [StoreOpener-deda1ce51fef7ef8d26da255318c55e4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C 2023-04-12 15:54:38,654 INFO [StoreOpener-deda1ce51fef7ef8d26da255318c55e4-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-12 15:54:38,655 INFO [StoreOpener-deda1ce51fef7ef8d26da255318c55e4-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region deda1ce51fef7ef8d26da255318c55e4 columnFamilyName C 2023-04-12 15:54:38,655 INFO [StoreOpener-deda1ce51fef7ef8d26da255318c55e4-1] regionserver.HStore(310): Store=deda1ce51fef7ef8d26da255318c55e4/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 15:54:38,655 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:38,656 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4 2023-04-12 15:54:38,656 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4 2023-04-12 15:54:38,657 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-12 15:54:38,658 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for deda1ce51fef7ef8d26da255318c55e4 2023-04-12 15:54:38,660 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-12 15:54:38,660 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened deda1ce51fef7ef8d26da255318c55e4; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69735221, jitterRate=0.03913576900959015}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-12 15:54:38,660 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for deda1ce51fef7ef8d26da255318c55e4: 2023-04-12 15:54:38,661 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4., pid=30, masterSystemTime=1681314878644 2023-04-12 15:54:38,662 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:38,662 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:38,663 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=deda1ce51fef7ef8d26da255318c55e4, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:38,663 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681314878662"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681314878662"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681314878662"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681314878662"}]},"ts":"1681314878662"} 2023-04-12 15:54:38,665 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=29 2023-04-12 15:54:38,665 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=29, state=SUCCESS; OpenRegionProcedure deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 in 172 msec 2023-04-12 15:54:38,666 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=28 2023-04-12 15:54:38,666 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=deda1ce51fef7ef8d26da255318c55e4, ASSIGN in 328 msec 2023-04-12 15:54:38,667 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-12 15:54:38,667 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681314878667"}]},"ts":"1681314878667"} 2023-04-12 15:54:38,668 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-12 15:54:38,670 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-12 15:54:38,671 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=28, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1620 sec 2023-04-12 15:54:39,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-12 15:54:39,615 INFO [Listener at localhost/43583] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 28 completed 2023-04-12 15:54:39,616 DEBUG [Listener at localhost/43583] zookeeper.ReadOnlyZKClient(139): Connect 0x5c124820 to 127.0.0.1:60154 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 15:54:39,621 DEBUG [Listener at localhost/43583] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@34b7fed0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 15:54:39,622 DEBUG [Listener at localhost/43583] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 15:54:39,623 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53000, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 15:54:39,624 DEBUG [Listener at localhost/43583] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-12 15:54:39,626 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37604, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-12 15:54:39,627 DEBUG [Listener at localhost/43583] zookeeper.ReadOnlyZKClient(139): Connect 0x3157d4df to 127.0.0.1:60154 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 15:54:39,631 DEBUG [Listener at localhost/43583] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7f4ad99d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 15:54:39,632 DEBUG [Listener at localhost/43583] zookeeper.ReadOnlyZKClient(139): Connect 0x3cf5bdb7 to 127.0.0.1:60154 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 15:54:39,637 DEBUG [Listener at localhost/43583] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@330b77da, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 15:54:39,638 DEBUG [Listener at localhost/43583] zookeeper.ReadOnlyZKClient(139): Connect 0x6919b2a0 to 127.0.0.1:60154 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 15:54:39,646 DEBUG [Listener at localhost/43583] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7810a231, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 15:54:39,646 DEBUG [Listener at localhost/43583] zookeeper.ReadOnlyZKClient(139): Connect 0x2ce5573e to 127.0.0.1:60154 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 15:54:39,649 DEBUG [Listener at localhost/43583] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@8cd7cb0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 15:54:39,650 DEBUG [Listener at localhost/43583] zookeeper.ReadOnlyZKClient(139): Connect 0x087b3095 to 127.0.0.1:60154 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 15:54:39,653 DEBUG [Listener at localhost/43583] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@ea1ca03, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 15:54:39,653 DEBUG [Listener at localhost/43583] zookeeper.ReadOnlyZKClient(139): Connect 0x433f251d to 127.0.0.1:60154 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 15:54:39,656 DEBUG [Listener at localhost/43583] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@434e0eab, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 15:54:39,657 DEBUG [Listener at localhost/43583] zookeeper.ReadOnlyZKClient(139): Connect 0x66ee0bd0 to 127.0.0.1:60154 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 15:54:39,667 DEBUG [Listener at localhost/43583] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5c43dddd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 15:54:39,667 DEBUG [Listener at localhost/43583] zookeeper.ReadOnlyZKClient(139): Connect 0x3b4658f8 to 127.0.0.1:60154 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 15:54:39,670 DEBUG [Listener at localhost/43583] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2de064cb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 15:54:39,670 DEBUG [Listener at localhost/43583] zookeeper.ReadOnlyZKClient(139): Connect 0x5d2b992d to 127.0.0.1:60154 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 15:54:39,674 DEBUG [Listener at localhost/43583] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@54a39be8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 15:54:39,674 DEBUG [Listener at localhost/43583] zookeeper.ReadOnlyZKClient(139): Connect 0x6a98265c to 127.0.0.1:60154 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 15:54:39,678 DEBUG [Listener at localhost/43583] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6bfb38d3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 15:54:39,682 DEBUG [hconnection-0x16ffdf38-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 15:54:39,682 DEBUG [hconnection-0x48df7f91-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 15:54:39,683 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:39,683 DEBUG [hconnection-0x6fdd3db0-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 15:54:39,684 DEBUG [hconnection-0x485a9fe4-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 15:54:39,684 DEBUG [hconnection-0x62fdc6b2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 15:54:39,684 DEBUG [hconnection-0x240f7ea8-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 15:54:39,684 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53012, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 15:54:39,685 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53032, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 15:54:39,688 DEBUG [hconnection-0x142d0c31-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 15:54:39,688 DEBUG [hconnection-0x77bb09b3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 15:54:39,689 DEBUG [hconnection-0x144969c5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 15:54:39,689 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53004, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 15:54:39,689 DEBUG [hconnection-0x7ff7763c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 15:54:39,690 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53028, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 15:54:39,690 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53002, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 15:54:39,690 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53014, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 15:54:39,691 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53064, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 15:54:39,694 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53050, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 15:54:39,696 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53040, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 15:54:39,697 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53078, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 15:54:39,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:39,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:39,697 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:39,697 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:39,697 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:39,698 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:39,698 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:39,698 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:39,701 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:39,701 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:39,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:39,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:39,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on deda1ce51fef7ef8d26da255318c55e4 2023-04-12 15:54:39,702 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing deda1ce51fef7ef8d26da255318c55e4 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-12 15:54:39,702 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK deda1ce51fef7ef8d26da255318c55e4, store=A 2023-04-12 15:54:39,702 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:39,702 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK deda1ce51fef7ef8d26da255318c55e4, store=B 2023-04-12 15:54:39,702 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:39,702 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK deda1ce51fef7ef8d26da255318c55e4, store=C 2023-04-12 15:54:39,703 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:39,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:39,703 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:39,703 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:39,703 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:39,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:39,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:39,704 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:39,706 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:39,706 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:39,706 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:39,706 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:39,706 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:39,708 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:39,708 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:39,708 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:39,708 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. started... 2023-04-12 15:54:39,708 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. as already flushing 2023-04-12 15:54:39,708 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:39,708 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:39,708 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:39,708 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:39,708 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:39,708 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:39,710 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:39,710 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:39,710 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:39,712 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:39,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:39,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:39,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:39,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:39,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:39,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:39,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:39,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:39,714 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:39,714 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c4b5416[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:39,714 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:39,714 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:39,716 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:39,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:39,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:39,716 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:39,716 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:39,716 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:39,716 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:39,716 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:39,717 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:39,717 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:39,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:39,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:39,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:39,717 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:39,718 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:39,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:39,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:39,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:39,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:39,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:39,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:39,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:39,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:39,720 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:39,720 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:39,720 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:39,720 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:39,720 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:39,720 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:39,723 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681314879716, max timestamp=1681314879722Num uniques 859; 2023-04-12 15:54:39,723 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681314879716, max timestamp=1681314879722Num uniques 859; 2023-04-12 15:54:39,723 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-12 15:54:39,724 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681314879716, max timestamp=1681314879722Num uniques 859; 2023-04-12 15:54:39,723 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-12 15:54:39,724 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-12 15:54:39,724 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:39,724 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:39,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:39,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:39,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:39,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:39,724 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:39,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:39,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:39,724 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:39,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:39,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:39,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:39,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:39,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:39,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:39,726 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:39,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:39,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:39,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:39,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:39,729 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:39,729 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:39,729 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:39,729 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:39,729 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:39,729 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:39,729 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:39,729 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:39,729 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:39,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:39,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:39,729 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:39,729 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:39,729 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:39,729 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:39,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:39,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:39,729 DEBUG [Thread-1647] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:39,730 DEBUG [Thread-1647] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:39,735 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:39,735 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:39,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53032 deadline: 1681314939732, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:39,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53040 deadline: 1681314939733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:39,737 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:39,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53078 deadline: 1681314939734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:39,737 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:39,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53012 deadline: 1681314939734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:39,737 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:39,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53014 deadline: 1681314939735, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:39,830 DEBUG [Thread-1647] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:39,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:39,832 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:39,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:39,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:39,833 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:39,833 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:39,833 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:39,834 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:39,834 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:39,834 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:39,836 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:39,836 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:39,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:39,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:39,836 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:39,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:39,836 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:39,837 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:39,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:39,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:39,837 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:39,837 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:39,837 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:39,838 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:39,838 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:39,838 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:39,838 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:39,838 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:39,838 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:39,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53032 deadline: 1681314939837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:39,838 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:39,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53040 deadline: 1681314939837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:39,839 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:39,839 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. started... 2023-04-12 15:54:39,839 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:39,839 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. as already flushing 2023-04-12 15:54:39,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53012 deadline: 1681314939838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:39,839 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:39,839 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:39,839 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:39,839 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:39,839 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:39,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53078 deadline: 1681314939838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:39,839 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:39,839 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:39,839 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:39,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53014 deadline: 1681314939838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:39,842 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:39,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:39,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:39,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:39,842 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:39,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:39,843 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:39,843 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:39,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:39,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:39,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:39,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:39,844 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:39,844 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@128846da[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:39,844 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:39,844 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:39,845 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:39,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:39,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:39,845 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:39,845 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:39,845 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:39,845 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:39,846 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:39,847 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:39,847 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:39,847 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:39,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:39,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:39,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:39,847 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:39,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:39,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:39,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:39,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:39,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:39,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:39,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:39,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:39,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:39,849 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:39,850 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:39,850 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:39,850 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:39,850 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:39,852 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:39,852 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:39,852 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:39,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:39,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:39,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:39,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:39,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:39,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:39,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:39,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:39,852 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:39,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:39,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:39,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:39,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:39,853 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:39,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:39,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:39,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:39,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:39,868 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:39,868 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:39,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:39,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:39,868 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:39,868 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:39,868 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:39,868 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:39,868 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:39,868 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:39,868 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:39,868 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:39,868 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:39,868 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:39,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:39,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:39,868 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:39,869 DEBUG [Thread-1647] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:39,869 DEBUG [Thread-1647] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:39,969 DEBUG [Thread-1647] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:39,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:39,971 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:39,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:39,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:39,973 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:39,973 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:39,973 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:39,973 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:39,973 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:39,973 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:39,975 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:39,975 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:39,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:39,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:39,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:39,976 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:39,976 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:39,976 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:39,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:39,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:39,976 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:39,984 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:39,984 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:39,984 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:39,984 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:39,984 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:39,988 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:39,988 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:39,988 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:39,988 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. started... 2023-04-12 15:54:39,988 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. as already flushing 2023-04-12 15:54:39,988 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:39,988 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:39,988 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:39,988 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:39,988 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:39,988 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:39,990 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:39,990 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:39,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:39,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:39,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:39,990 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:39,990 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:39,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:39,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:39,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:39,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:39,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:39,992 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:39,992 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75bb382b[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:39,992 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:39,992 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:39,994 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:39,994 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:39,994 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:39,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:39,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:39,994 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:39,994 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:39,994 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:39,996 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:39,996 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:39,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:39,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:39,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:39,996 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:39,997 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:39,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:39,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:39,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:39,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:39,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:39,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:39,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:39,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:39,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:39,999 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:39,999 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:39,999 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:39,999 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:39,999 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:40,001 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:40,001 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:40,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:40,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:40,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:40,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:40,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:40,001 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:40,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:40,001 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:40,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:40,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:40,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:40,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:40,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:40,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:40,002 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:40,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:40,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:40,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:40,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:40,005 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:40,005 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:40,005 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:40,005 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:40,005 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:40,005 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:40,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:40,005 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:40,005 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:40,005 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:40,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:40,005 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:40,005 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:40,005 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:40,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:40,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:40,006 DEBUG [Thread-1647] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:40,006 DEBUG [Thread-1647] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:40,006 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:40,016 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-04-12 15:54:40,042 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:40,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53012 deadline: 1681314940040, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:40,042 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:40,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53032 deadline: 1681314940040, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:40,043 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:40,043 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:40,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53040 deadline: 1681314940041, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:40,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53014 deadline: 1681314940041, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:40,043 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:40,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53078 deadline: 1681314940041, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:40,106 DEBUG [Thread-1647] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:40,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:40,108 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:40,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:40,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:40,109 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:40,109 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:40,110 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:40,110 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:40,110 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:40,110 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:40,111 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:40,111 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:40,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:40,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:40,111 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:40,112 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:40,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:40,112 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:40,113 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=11 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/A/b99771cc50114253849cbfb2756ffd83 2023-04-12 15:54:40,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:40,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:40,113 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:40,113 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:40,113 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:40,114 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:40,114 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:40,114 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:40,114 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:40,114 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:40,114 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:40,114 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. started... 2023-04-12 15:54:40,114 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. as already flushing 2023-04-12 15:54:40,114 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:40,114 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:40,114 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:40,114 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:40,114 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:40,114 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:40,116 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:40,117 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:40,117 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:40,117 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:40,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:40,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:40,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:40,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:40,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:40,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:40,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:40,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:40,119 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:40,119 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@512be6eb[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:40,119 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:40,120 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:40,121 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:40,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:40,121 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:40,122 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:40,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:40,122 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:40,122 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:40,122 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:40,123 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:40,123 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:40,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:40,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:40,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:40,123 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:40,123 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:40,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:40,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:40,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:40,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:40,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:40,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:40,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:40,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:40,126 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:40,126 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:40,126 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:40,126 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:40,126 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:40,126 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:40,128 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:40,128 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:40,128 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:40,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:40,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:40,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:40,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:40,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:40,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:40,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:40,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:40,129 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=11 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/B/a2523d426d53476cae46976432ba478f 2023-04-12 15:54:40,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:40,129 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:40,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:40,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:40,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:40,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:40,131 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:40,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:40,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:40,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:40,134 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:40,134 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:40,134 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:40,134 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:40,134 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:40,134 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:40,134 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:40,134 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:40,134 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:40,134 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:40,135 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:40,135 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:40,135 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:40,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:40,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:40,135 DEBUG [Thread-1647] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:40,135 DEBUG [Thread-1647] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:40,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:40,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:40,235 DEBUG [Thread-1647] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:40,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:40,237 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:40,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:40,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:40,239 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:40,239 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:40,239 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:40,239 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:40,239 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:40,239 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:40,240 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:40,240 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:40,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:40,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:40,241 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:40,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:40,241 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:40,241 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:40,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:40,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:40,241 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:40,241 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:40,242 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:40,242 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:40,242 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:40,242 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:40,242 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:40,242 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:40,242 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:40,242 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. started... 2023-04-12 15:54:40,242 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. as already flushing 2023-04-12 15:54:40,242 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:40,242 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:40,242 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:40,242 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:40,242 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:40,242 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:40,245 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:40,245 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:40,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:40,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:40,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:40,245 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:40,246 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:40,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:40,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:40,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:40,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:40,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:40,247 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:40,247 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3726f33c[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:40,247 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:40,247 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:40,248 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:40,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:40,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:40,248 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:40,248 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:40,248 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:40,248 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:40,248 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:40,250 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:40,250 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:40,250 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:40,250 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:40,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:40,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:40,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:40,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:40,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:40,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:40,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:40,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:40,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:40,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:40,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:40,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:40,252 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:40,252 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:40,252 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:40,252 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:40,252 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:40,254 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:40,254 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:40,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:40,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:40,254 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:40,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:40,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:40,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:40,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:40,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:40,254 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:40,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:40,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:40,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:40,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:40,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:40,256 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:40,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:40,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:40,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:40,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:40,258 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:40,258 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:40,258 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:40,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:40,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:40,259 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:40,259 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:40,259 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:40,259 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:40,259 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:40,259 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:40,259 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:40,259 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:40,259 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:40,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:40,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:40,259 DEBUG [Thread-1647] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:40,259 DEBUG [Thread-1647] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:40,259 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:40,345 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:40,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53040 deadline: 1681314940344, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:40,346 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:40,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53014 deadline: 1681314940345, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:40,346 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:40,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53078 deadline: 1681314940345, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:40,346 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:40,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53012 deadline: 1681314940345, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:40,346 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:40,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53032 deadline: 1681314940345, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:40,359 DEBUG [Thread-1647] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:40,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:40,361 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:40,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:40,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:40,365 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:40,365 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:40,365 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:40,365 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:40,365 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:40,365 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:40,366 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:40,366 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:40,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:40,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:40,367 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:40,367 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:40,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:40,367 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:40,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:40,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:40,368 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:40,368 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:40,368 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:40,368 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:40,368 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:40,368 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:40,368 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:40,368 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:40,368 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:40,368 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. started... 2023-04-12 15:54:40,368 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. as already flushing 2023-04-12 15:54:40,368 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:40,368 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:40,369 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:40,369 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:40,369 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:40,369 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:40,376 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:40,376 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:40,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:40,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:40,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:40,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:40,377 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:40,377 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:40,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:40,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:40,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:40,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:40,379 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:40,379 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:40,379 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:40,379 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6748dd94[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:40,380 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:40,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:40,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:40,380 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:40,380 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:40,380 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:40,380 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:40,380 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:40,381 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:40,381 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:40,381 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:40,382 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:40,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:40,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:40,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:40,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:40,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:40,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:40,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:40,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:40,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:40,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:40,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:40,385 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:40,385 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:40,385 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:40,385 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:40,385 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:40,385 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:40,388 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:40,388 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:40,388 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:40,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:40,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:40,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:40,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:40,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:40,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:40,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:40,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:40,388 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:40,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:40,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:40,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:40,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:40,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:40,390 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:40,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:40,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:40,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:40,393 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:40,393 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:40,393 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:40,393 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:40,393 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:40,393 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:40,393 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:40,393 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:40,393 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:40,393 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:40,393 DEBUG [Thread-1647] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:40,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:40,393 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:40,393 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:40,393 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:40,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:40,393 DEBUG [Thread-1647] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:40,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:40,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:40,494 DEBUG [Thread-1647] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:40,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:40,496 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:40,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:40,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:40,497 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:40,497 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:40,497 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:40,497 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:40,498 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:40,498 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:40,499 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:40,499 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:40,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:40,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:40,499 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:40,499 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:40,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:40,500 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:40,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:40,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:40,500 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:40,500 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:40,500 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:40,501 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:40,501 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:40,501 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:40,501 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:40,501 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:40,501 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:40,501 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. started... 2023-04-12 15:54:40,501 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. as already flushing 2023-04-12 15:54:40,501 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:40,501 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:40,501 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:40,501 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:40,501 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:40,501 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:40,502 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:40,502 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:40,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:40,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:40,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:40,503 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:40,503 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:40,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:40,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:40,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:40,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:40,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:40,504 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:40,504 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:40,504 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b1edb36[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:40,504 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:40,506 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:40,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:40,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:40,507 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:40,507 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:40,507 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:40,507 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:40,507 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:40,508 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:40,508 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:40,508 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:40,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:40,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:40,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:40,508 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:40,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:40,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:40,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:40,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:40,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:40,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:40,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:40,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:40,511 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:40,511 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:40,511 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:40,511 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:40,511 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:40,511 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:40,512 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:40,512 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:40,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:40,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:40,512 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:40,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:40,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:40,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:40,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:40,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:40,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:40,513 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:40,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:40,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:40,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:40,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:40,515 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:40,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:40,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:40,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:40,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:40,517 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:40,517 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:40,517 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:40,517 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:40,517 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:40,517 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:40,517 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:40,517 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:40,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:40,517 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:40,517 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:40,518 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:40,518 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:40,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:40,518 DEBUG [Thread-1647] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:40,518 DEBUG [Thread-1647] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:40,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:40,518 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:40,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:40,542 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=11 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/C/f00576ffc88f428cb64b448151eb504b 2023-04-12 15:54:40,546 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/A/b99771cc50114253849cbfb2756ffd83 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/b99771cc50114253849cbfb2756ffd83 2023-04-12 15:54:40,550 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/b99771cc50114253849cbfb2756ffd83, entries=150, sequenceid=11, filesize=11.6 K 2023-04-12 15:54:40,551 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/B/a2523d426d53476cae46976432ba478f as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/a2523d426d53476cae46976432ba478f 2023-04-12 15:54:40,554 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/a2523d426d53476cae46976432ba478f, entries=150, sequenceid=11, filesize=11.6 K 2023-04-12 15:54:40,555 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/C/f00576ffc88f428cb64b448151eb504b as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/f00576ffc88f428cb64b448151eb504b 2023-04-12 15:54:40,565 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/f00576ffc88f428cb64b448151eb504b, entries=150, sequenceid=11, filesize=11.6 K 2023-04-12 15:54:40,566 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for deda1ce51fef7ef8d26da255318c55e4 in 864ms, sequenceid=11, compaction requested=false 2023-04-12 15:54:40,566 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for deda1ce51fef7ef8d26da255318c55e4: 2023-04-12 15:54:40,618 DEBUG [Thread-1647] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:40,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:40,620 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:40,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:40,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:40,624 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:40,624 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:40,624 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:40,624 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:40,624 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:40,624 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:40,627 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:40,627 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:40,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:40,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:40,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:40,627 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:40,627 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:40,628 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:40,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:40,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:40,628 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:40,628 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:40,628 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:40,628 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:40,628 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:40,629 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:40,629 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:40,629 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:40,629 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:40,629 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. started... 2023-04-12 15:54:40,629 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing deda1ce51fef7ef8d26da255318c55e4 3/3 column families, dataSize=134.18 KB heapSize=375.61 KB 2023-04-12 15:54:40,629 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK deda1ce51fef7ef8d26da255318c55e4, store=A 2023-04-12 15:54:40,629 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:54:40,629 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK deda1ce51fef7ef8d26da255318c55e4, store=B 2023-04-12 15:54:40,629 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:54:40,629 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK deda1ce51fef7ef8d26da255318c55e4, store=C 2023-04-12 15:54:40,629 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:54:40,638 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=46 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/A/386394a2b715447892aee082948007ac 2023-04-12 15:54:40,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on deda1ce51fef7ef8d26da255318c55e4 2023-04-12 15:54:40,848 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. as already flushing 2023-04-12 15:54:40,855 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:40,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53040 deadline: 1681314940852, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:40,856 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:40,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53012 deadline: 1681314940853, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:40,857 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:40,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53014 deadline: 1681314940854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:40,858 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:40,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53078 deadline: 1681314940854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:40,859 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:40,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53032 deadline: 1681314940855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:40,957 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:40,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53040 deadline: 1681314940956, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:40,957 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:40,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53012 deadline: 1681314940957, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:40,960 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:40,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53014 deadline: 1681314940958, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:40,960 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:40,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53078 deadline: 1681314940958, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:40,961 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:40,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53032 deadline: 1681314940960, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:41,050 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=46 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/B/c868446dfeb749948cabcda3c1f9c271 2023-04-12 15:54:41,159 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:41,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53040 deadline: 1681314941158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:41,161 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:41,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53012 deadline: 1681314941159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:41,163 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:41,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53032 deadline: 1681314941162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:41,163 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:41,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53014 deadline: 1681314941162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:41,163 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:41,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53078 deadline: 1681314941162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:41,462 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:41,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53012 deadline: 1681314941462, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:41,463 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:41,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53040 deadline: 1681314941462, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:41,464 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=46 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/C/26c610ee0ad54ceb809f277f972104d4 2023-04-12 15:54:41,467 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:41,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53032 deadline: 1681314941464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:41,468 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:41,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53014 deadline: 1681314941467, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:41,468 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:41,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53078 deadline: 1681314941467, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:41,471 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/A/386394a2b715447892aee082948007ac as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/386394a2b715447892aee082948007ac 2023-04-12 15:54:41,474 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/386394a2b715447892aee082948007ac, entries=150, sequenceid=46, filesize=11.6 K 2023-04-12 15:54:41,475 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/B/c868446dfeb749948cabcda3c1f9c271 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/c868446dfeb749948cabcda3c1f9c271 2023-04-12 15:54:41,479 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/c868446dfeb749948cabcda3c1f9c271, entries=150, sequenceid=46, filesize=11.6 K 2023-04-12 15:54:41,479 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/C/26c610ee0ad54ceb809f277f972104d4 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/26c610ee0ad54ceb809f277f972104d4 2023-04-12 15:54:41,483 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/26c610ee0ad54ceb809f277f972104d4, entries=150, sequenceid=46, filesize=11.6 K 2023-04-12 15:54:41,484 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~375.56 KB/384576, currentSize=53.67 KB/54960 for deda1ce51fef7ef8d26da255318c55e4 in 855ms, sequenceid=46, compaction requested=false 2023-04-12 15:54:41,484 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for deda1ce51fef7ef8d26da255318c55e4: 2023-04-12 15:54:41,484 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:41,484 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:41,484 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:41,484 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:41,484 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:41,484 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:41,486 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:41,486 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:41,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:41,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:41,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:41,486 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:41,486 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:41,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:41,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:41,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:41,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:41,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:41,487 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:41,487 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@458c6000[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:41,487 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:41,488 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:41,489 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:41,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:41,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:41,490 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:41,490 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:41,490 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:41,490 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:41,490 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:41,491 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:41,491 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:41,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:41,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:41,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:41,491 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:41,491 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:41,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:41,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:41,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:41,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:41,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:41,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:41,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:41,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:41,494 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:41,494 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:41,494 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:41,494 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:41,494 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:41,494 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:41,496 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:41,496 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:41,496 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:41,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:41,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:41,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:41,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:41,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:41,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:41,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:41,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:41,497 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:41,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:41,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:41,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:41,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:41,498 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:41,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:41,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:41,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:41,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:41,501 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:41,501 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:41,501 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:41,501 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:41,501 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:41,501 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:41,501 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:41,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:41,501 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:41,501 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:41,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:41,501 DEBUG [Thread-1647] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:41,501 DEBUG [Thread-1647] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:41,501 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:41,501 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:41,501 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:41,501 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:41,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:41,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:41,601 DEBUG [Thread-1647] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:41,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:41,604 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:41,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:41,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:41,605 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:41,605 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:41,605 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:41,605 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:41,605 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:41,605 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:41,608 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:41,608 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:41,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:41,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:41,608 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:41,608 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:41,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:41,608 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:41,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:41,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:41,609 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:41,609 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:41,609 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:41,609 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:41,609 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:41,609 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:41,609 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:41,609 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:41,609 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:41,609 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. started... 2023-04-12 15:54:41,610 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing deda1ce51fef7ef8d26da255318c55e4 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-12 15:54:41,610 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK deda1ce51fef7ef8d26da255318c55e4, store=A 2023-04-12 15:54:41,610 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:41,610 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK deda1ce51fef7ef8d26da255318c55e4, store=B 2023-04-12 15:54:41,610 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:41,610 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK deda1ce51fef7ef8d26da255318c55e4, store=C 2023-04-12 15:54:41,610 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:41,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on deda1ce51fef7ef8d26da255318c55e4 2023-04-12 15:54:41,968 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. as already flushing 2023-04-12 15:54:41,986 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681314881980, max timestamp=1681314881985Num uniques 859; 2023-04-12 15:54:41,987 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-12 15:54:41,987 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681314881980, max timestamp=1681314881985Num uniques 859; 2023-04-12 15:54:41,987 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-12 15:54:41,987 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681314881980, max timestamp=1681314881985Num uniques 859; 2023-04-12 15:54:41,987 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-12 15:54:42,001 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:42,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53012 deadline: 1681314941998, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:42,004 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:42,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53014 deadline: 1681314941999, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:42,004 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:42,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53078 deadline: 1681314942000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:42,005 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:42,005 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:42,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53040 deadline: 1681314942001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:42,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53032 deadline: 1681314942001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:42,029 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=57 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/A/9ac3c8a644cc4147bdd1e1a75edd0021 2023-04-12 15:54:42,104 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:42,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53012 deadline: 1681314942102, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:42,107 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:42,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53014 deadline: 1681314942104, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:42,107 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:42,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53078 deadline: 1681314942105, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:42,108 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:42,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53032 deadline: 1681314942106, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:42,108 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:42,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53040 deadline: 1681314942106, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:42,307 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:42,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53012 deadline: 1681314942306, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:42,310 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:42,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53014 deadline: 1681314942308, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:42,310 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:42,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53078 deadline: 1681314942309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:42,312 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:42,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53032 deadline: 1681314942310, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:42,312 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:42,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53040 deadline: 1681314942310, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:42,354 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-12 15:54:42,439 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=57 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/B/690c69166d0c44b2ba302fa0c8dce93f 2023-04-12 15:54:42,609 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:42,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53012 deadline: 1681314942608, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:42,612 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:42,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53014 deadline: 1681314942611, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:42,615 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:42,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53078 deadline: 1681314942613, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:42,616 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:42,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53032 deadline: 1681314942613, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:42,616 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:42,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53040 deadline: 1681314942614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:42,849 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=57 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/C/9e703618cbd94eeda86c14a7ff44ae98 2023-04-12 15:54:42,854 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/A/9ac3c8a644cc4147bdd1e1a75edd0021 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/9ac3c8a644cc4147bdd1e1a75edd0021 2023-04-12 15:54:42,858 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/9ac3c8a644cc4147bdd1e1a75edd0021, entries=150, sequenceid=57, filesize=11.6 K 2023-04-12 15:54:42,859 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/B/690c69166d0c44b2ba302fa0c8dce93f as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/690c69166d0c44b2ba302fa0c8dce93f 2023-04-12 15:54:42,865 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/690c69166d0c44b2ba302fa0c8dce93f, entries=150, sequenceid=57, filesize=11.6 K 2023-04-12 15:54:42,868 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/C/9e703618cbd94eeda86c14a7ff44ae98 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/9e703618cbd94eeda86c14a7ff44ae98 2023-04-12 15:54:42,873 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/9e703618cbd94eeda86c14a7ff44ae98, entries=150, sequenceid=57, filesize=11.6 K 2023-04-12 15:54:42,874 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for deda1ce51fef7ef8d26da255318c55e4 in 1263ms, sequenceid=57, compaction requested=true 2023-04-12 15:54:42,874 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for deda1ce51fef7ef8d26da255318c55e4: 2023-04-12 15:54:42,874 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:42,874 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:42,874 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:42,874 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:42,880 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:42,880 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:42,881 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:42,881 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:42,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:42,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:42,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:42,882 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:42,882 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:42,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:42,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:42,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:42,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:42,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:42,883 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:42,883 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4533c49e[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:42,883 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:42,883 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:42,887 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:42,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:42,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:42,887 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:42,887 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:42,887 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:42,887 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:42,888 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:42,889 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:42,889 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:42,889 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:42,889 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:42,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:42,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:42,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:42,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:42,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:42,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:42,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:42,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:42,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:42,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:42,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:42,892 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:42,892 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:42,892 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:42,892 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:42,892 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:42,892 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:42,893 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:42,893 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:42,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:42,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:42,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:42,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:42,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:42,893 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:42,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:42,894 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:42,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:42,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:42,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:42,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:42,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:42,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:42,895 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:42,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:42,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:42,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:42,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:42,899 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:42,899 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:42,899 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:42,899 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:42,899 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:42,899 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:42,899 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:42,899 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:42,899 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:42,899 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:42,899 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:42,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:42,899 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:42,900 DEBUG [Thread-1647] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:42,899 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:42,900 DEBUG [Thread-1647] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:42,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:42,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:42,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:43,000 DEBUG [Thread-1647] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:43,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:43,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:43,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:43,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:43,003 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:43,003 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:43,003 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:43,003 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:43,003 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:43,004 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:43,005 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:43,005 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:43,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:43,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:43,005 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:43,005 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:43,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:43,005 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:43,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:43,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:43,006 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:43,006 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:43,006 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:43,006 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:43,006 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:43,006 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:43,006 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:43,007 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:43,007 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:43,007 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. started... 2023-04-12 15:54:43,007 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing deda1ce51fef7ef8d26da255318c55e4 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-12 15:54:43,007 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK deda1ce51fef7ef8d26da255318c55e4, store=A 2023-04-12 15:54:43,007 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:54:43,007 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK deda1ce51fef7ef8d26da255318c55e4, store=B 2023-04-12 15:54:43,007 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:54:43,007 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK deda1ce51fef7ef8d26da255318c55e4, store=C 2023-04-12 15:54:43,007 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:54:43,024 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=93 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/A/05aa95731e504028a8a31f024a0756e3 2023-04-12 15:54:43,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on deda1ce51fef7ef8d26da255318c55e4 2023-04-12 15:54:43,116 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. as already flushing 2023-04-12 15:54:43,126 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:43,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53040 deadline: 1681314943122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:43,126 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:43,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53012 deadline: 1681314943122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:43,128 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:43,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53078 deadline: 1681314943125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:43,128 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:43,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53014 deadline: 1681314943125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:43,129 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:43,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53032 deadline: 1681314943126, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:43,229 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:43,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53012 deadline: 1681314943227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:43,229 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:43,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53040 deadline: 1681314943227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:43,231 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:43,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53078 deadline: 1681314943229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:43,231 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:43,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53014 deadline: 1681314943229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:43,231 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:43,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53032 deadline: 1681314943230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:43,433 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:43,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53012 deadline: 1681314943430, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:43,433 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:43,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53040 deadline: 1681314943430, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:43,433 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:43,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53078 deadline: 1681314943432, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:43,434 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:43,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53032 deadline: 1681314943432, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:43,435 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:43,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53014 deadline: 1681314943433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:43,453 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=93 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/B/c01ff948b9e242bb828790d2f8b52ac5 2023-04-12 15:54:43,468 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=93 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/C/b2ab52c67a604d3ca27b2f62d5084af9 2023-04-12 15:54:43,472 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/A/05aa95731e504028a8a31f024a0756e3 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/05aa95731e504028a8a31f024a0756e3 2023-04-12 15:54:43,475 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/05aa95731e504028a8a31f024a0756e3, entries=150, sequenceid=93, filesize=11.6 K 2023-04-12 15:54:43,476 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/B/c01ff948b9e242bb828790d2f8b52ac5 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/c01ff948b9e242bb828790d2f8b52ac5 2023-04-12 15:54:43,482 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/c01ff948b9e242bb828790d2f8b52ac5, entries=150, sequenceid=93, filesize=11.6 K 2023-04-12 15:54:43,483 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/C/b2ab52c67a604d3ca27b2f62d5084af9 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/b2ab52c67a604d3ca27b2f62d5084af9 2023-04-12 15:54:43,488 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/b2ab52c67a604d3ca27b2f62d5084af9, entries=150, sequenceid=93, filesize=11.6 K 2023-04-12 15:54:43,489 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for deda1ce51fef7ef8d26da255318c55e4 in 482ms, sequenceid=93, compaction requested=true 2023-04-12 15:54:43,489 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for deda1ce51fef7ef8d26da255318c55e4: 2023-04-12 15:54:43,489 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:43,489 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:43,489 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:43,489 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:43,489 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:43,490 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:43,491 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:43,491 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:43,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:43,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:43,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:43,491 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:43,491 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:43,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:43,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:43,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:43,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:43,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:43,493 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:43,493 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@643615d1[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:43,493 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:43,493 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:43,494 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:43,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:43,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:43,494 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:43,494 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:43,494 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:43,494 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:43,494 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:43,499 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:43,499 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:43,499 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:43,499 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:43,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:43,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:43,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:43,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:43,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:43,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:43,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:43,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:43,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:43,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:43,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:43,502 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:43,502 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:43,502 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:43,502 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:43,502 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:43,502 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:43,503 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:43,503 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:43,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:43,504 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:43,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:43,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:43,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:43,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:43,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:43,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:43,504 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:43,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:43,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:43,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:43,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:43,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:43,505 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:43,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:43,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:43,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:43,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:43,507 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:43,507 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:43,507 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:43,507 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:43,507 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:43,507 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:43,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:43,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:43,507 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:43,507 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:43,507 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:43,508 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:43,508 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:43,508 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:43,508 DEBUG [Thread-1647] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:43,508 DEBUG [Thread-1647] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:43,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:43,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:43,508 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:43,608 DEBUG [Thread-1647] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:43,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:43,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:43,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:43,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:43,611 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:43,612 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:43,612 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:43,612 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:43,612 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:43,612 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:43,613 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:43,613 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:43,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:43,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:43,614 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:43,614 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:43,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:43,614 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:43,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:43,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:43,614 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:43,615 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:43,615 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:43,615 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:43,615 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:43,615 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:43,615 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:43,615 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:43,615 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:43,615 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. started... 2023-04-12 15:54:43,615 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing deda1ce51fef7ef8d26da255318c55e4 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-12 15:54:43,616 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK deda1ce51fef7ef8d26da255318c55e4, store=A 2023-04-12 15:54:43,616 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:43,616 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK deda1ce51fef7ef8d26da255318c55e4, store=B 2023-04-12 15:54:43,616 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:43,616 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK deda1ce51fef7ef8d26da255318c55e4, store=C 2023-04-12 15:54:43,616 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:43,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on deda1ce51fef7ef8d26da255318c55e4 2023-04-12 15:54:43,737 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. as already flushing 2023-04-12 15:54:43,755 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681314883743, max timestamp=1681314883753Num uniques 859; 2023-04-12 15:54:43,755 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681314883743, max timestamp=1681314883753Num uniques 859; 2023-04-12 15:54:43,755 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681314883743, max timestamp=1681314883753Num uniques 859; 2023-04-12 15:54:43,755 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-12 15:54:43,755 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-12 15:54:43,755 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-12 15:54:43,768 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:43,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53040 deadline: 1681314943763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:43,768 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:43,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53032 deadline: 1681314943765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:43,768 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:43,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53014 deadline: 1681314943766, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:43,771 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:43,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53078 deadline: 1681314943767, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:43,771 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:43,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53012 deadline: 1681314943768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:43,871 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:43,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53040 deadline: 1681314943869, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:43,871 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:43,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53032 deadline: 1681314943869, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:43,872 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:43,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53014 deadline: 1681314943870, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:43,874 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:43,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53078 deadline: 1681314943872, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:43,874 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:43,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53012 deadline: 1681314943872, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:44,026 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=104 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/A/45de1a82700642a5b0231e12f3276cb9 2023-04-12 15:54:44,075 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:44,075 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:44,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53040 deadline: 1681314944072, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:44,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53032 deadline: 1681314944072, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:44,075 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:44,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53014 deadline: 1681314944073, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:44,080 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:44,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53078 deadline: 1681314944076, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:44,080 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:44,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53012 deadline: 1681314944076, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:44,378 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:44,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53040 deadline: 1681314944376, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:44,378 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:44,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53014 deadline: 1681314944376, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:44,380 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:44,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53032 deadline: 1681314944378, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:44,384 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:44,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53078 deadline: 1681314944382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:44,384 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:44,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53012 deadline: 1681314944383, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:44,437 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=104 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/B/869399d589004bb1bec558eee7c12205 2023-04-12 15:54:44,862 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=104 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/C/1f8ae434e8024e1cadd722956b5368bd 2023-04-12 15:54:44,867 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/A/45de1a82700642a5b0231e12f3276cb9 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/45de1a82700642a5b0231e12f3276cb9 2023-04-12 15:54:44,870 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/45de1a82700642a5b0231e12f3276cb9, entries=150, sequenceid=104, filesize=11.6 K 2023-04-12 15:54:44,871 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/B/869399d589004bb1bec558eee7c12205 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/869399d589004bb1bec558eee7c12205 2023-04-12 15:54:44,875 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/869399d589004bb1bec558eee7c12205, entries=150, sequenceid=104, filesize=11.6 K 2023-04-12 15:54:44,875 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/C/1f8ae434e8024e1cadd722956b5368bd as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/1f8ae434e8024e1cadd722956b5368bd 2023-04-12 15:54:44,879 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/1f8ae434e8024e1cadd722956b5368bd, entries=150, sequenceid=104, filesize=11.6 K 2023-04-12 15:54:44,879 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for deda1ce51fef7ef8d26da255318c55e4 in 1264ms, sequenceid=104, compaction requested=true 2023-04-12 15:54:44,879 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for deda1ce51fef7ef8d26da255318c55e4: 2023-04-12 15:54:44,879 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:44,879 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:44,880 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:44,880 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:44,880 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:44,880 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:44,881 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:44,881 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:44,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:44,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:44,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:44,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:44,882 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:44,882 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:44,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:44,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on deda1ce51fef7ef8d26da255318c55e4 2023-04-12 15:54:44,882 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing deda1ce51fef7ef8d26da255318c55e4 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-12 15:54:44,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:44,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:44,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:44,883 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK deda1ce51fef7ef8d26da255318c55e4, store=A 2023-04-12 15:54:44,884 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:54:44,884 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK deda1ce51fef7ef8d26da255318c55e4, store=B 2023-04-12 15:54:44,884 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:54:44,884 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK deda1ce51fef7ef8d26da255318c55e4, store=C 2023-04-12 15:54:44,884 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:54:44,884 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:44,884 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59cca50a[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:44,884 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:44,884 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:44,885 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:44,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:44,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:44,885 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:44,885 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:44,885 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:44,886 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:44,886 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:44,887 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:44,887 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:44,887 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:44,887 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:44,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:44,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:44,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:44,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:44,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:44,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:44,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:44,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:44,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:44,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:44,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:44,891 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:44,891 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:44,891 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:44,891 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:44,891 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:44,891 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:44,892 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=141 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/A/212f2c600b0a45ae8c36202a72f0fc40 2023-04-12 15:54:44,894 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:44,894 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:44,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:44,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:44,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:44,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:44,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:44,894 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:44,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:44,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:44,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:44,895 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:44,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:44,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:44,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:44,895 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:44,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53014 deadline: 1681314944890, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:44,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:44,896 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:44,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53032 deadline: 1681314944890, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:44,896 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:44,896 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:44,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53078 deadline: 1681314944891, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:44,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:44,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:44,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:44,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:44,900 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:44,900 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:44,900 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:44,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:44,900 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:44,900 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:44,900 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:44,900 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:44,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53012 deadline: 1681314944894, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:44,900 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:44,900 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:44,900 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:44,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:44,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53040 deadline: 1681314944895, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:44,900 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:44,900 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:44,901 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:44,901 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:44,901 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:44,901 DEBUG [Thread-1647] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:44,901 DEBUG [Thread-1647] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:44,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:44,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:44,999 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:44,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53014 deadline: 1681314944997, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:44,999 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:45,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53032 deadline: 1681314944997, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,000 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:45,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53078 deadline: 1681314944997, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,001 DEBUG [Thread-1647] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:45,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:45,003 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:45,004 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:45,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53012 deadline: 1681314945001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,004 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:45,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53040 deadline: 1681314945001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:45,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:45,006 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:45,006 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:45,006 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:45,006 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:45,006 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:45,006 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:45,007 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:45,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:45,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:45,007 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:45,008 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,008 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:45,008 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:45,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:45,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:45,008 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:45,009 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:45,009 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:45,009 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:45,009 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:45,009 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:45,009 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:45,009 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:45,009 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:45,009 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. started... 2023-04-12 15:54:45,009 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. as already flushing 2023-04-12 15:54:45,009 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:45,009 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:45,009 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:45,009 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,009 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:45,009 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:45,011 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,011 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:45,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:45,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:45,011 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:45,011 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:45,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:45,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:45,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:45,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:45,012 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:45,013 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f665253[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:45,013 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:45,013 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:45,015 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:45,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:45,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:45,015 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:45,015 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,015 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:45,015 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:45,015 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:45,019 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:45,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:45,019 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:45,019 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:45,019 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:45,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:45,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:45,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:45,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:45,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:45,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:45,022 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:45,022 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:45,022 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:45,022 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:45,022 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:45,022 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:45,023 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:45,023 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:45,023 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:45,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:45,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:45,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:45,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:45,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:45,024 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:45,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:45,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:45,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:45,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:45,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:45,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:45,025 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:45,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:45,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,027 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,027 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:45,027 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:45,027 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:45,027 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:45,027 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:45,027 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:45,027 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:45,027 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:45,028 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:45,028 DEBUG [Thread-1647] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:45,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:45,028 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:45,028 DEBUG [Thread-1647] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:45,028 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:45,028 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:45,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:45,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:45,128 DEBUG [Thread-1647] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:45,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:45,130 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:45,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:45,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:45,131 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:45,132 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:45,132 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:45,132 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:45,132 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:45,132 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:45,134 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,134 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:45,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:45,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:45,134 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,134 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:45,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:45,135 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:45,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:45,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:45,135 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:45,136 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:45,136 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:45,136 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:45,136 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:45,136 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:45,136 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:45,136 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:45,136 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:45,136 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. started... 2023-04-12 15:54:45,136 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. as already flushing 2023-04-12 15:54:45,136 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:45,136 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:45,136 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:45,136 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,136 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:45,136 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:45,138 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,138 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:45,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:45,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:45,138 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:45,138 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:45,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:45,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:45,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:45,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:45,140 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:45,140 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24aa6e15[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:45,140 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:45,140 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:45,141 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:45,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:45,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:45,141 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:45,141 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:45,141 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:45,141 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,141 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:45,142 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:45,142 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:45,142 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,142 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:45,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:45,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:45,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:45,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:45,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:45,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:45,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:45,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:45,145 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:45,145 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:45,145 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:45,145 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:45,145 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:45,145 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:45,147 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:45,147 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:45,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:45,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:45,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:45,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:45,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:45,147 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:45,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:45,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:45,148 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:45,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:45,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:45,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:45,149 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:45,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:45,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:45,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,151 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:45,151 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:45,151 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:45,151 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:45,151 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:45,151 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:45,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:45,151 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:45,152 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:45,152 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:45,152 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,152 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:45,152 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:45,152 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:45,152 DEBUG [Thread-1647] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:45,152 DEBUG [Thread-1647] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:45,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:45,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:45,203 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:45,203 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:45,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53014 deadline: 1681314945200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53032 deadline: 1681314945201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,204 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:45,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53078 deadline: 1681314945201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,208 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:45,208 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:45,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53012 deadline: 1681314945205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53040 deadline: 1681314945205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,252 DEBUG [Thread-1647] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:45,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:45,254 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:45,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:45,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:45,256 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:45,256 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:45,256 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:45,256 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:45,256 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:45,256 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:45,257 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:45,257 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:45,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:45,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:45,258 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,258 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:45,258 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:45,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:45,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:45,259 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:45,259 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:45,259 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:45,259 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:45,259 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:45,259 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:45,259 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:45,259 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:45,260 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:45,260 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. started... 2023-04-12 15:54:45,260 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. as already flushing 2023-04-12 15:54:45,260 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:45,260 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:45,260 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:45,260 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,260 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:45,260 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:45,261 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,261 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:45,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:45,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:45,262 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:45,262 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:45,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:45,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:45,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:45,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:45,263 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:45,263 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70666fcc[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:45,263 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:45,263 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:45,268 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:45,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:45,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:45,269 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:45,269 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:45,269 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:45,269 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,269 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:45,270 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,270 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:45,270 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:45,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:45,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:45,270 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:45,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:45,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:45,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:45,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:45,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:45,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:45,272 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:45,272 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:45,272 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:45,272 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:45,272 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:45,272 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:45,273 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:45,273 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:45,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:45,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:45,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:45,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:45,273 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:45,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:45,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:45,274 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:45,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:45,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:45,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:45,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:45,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:45,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,275 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:45,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:45,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,277 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:45,277 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:45,277 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:45,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:45,277 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:45,277 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:45,277 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:45,278 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:45,278 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:45,278 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,278 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:45,278 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:45,278 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:45,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:45,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:45,278 DEBUG [Thread-1647] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:45,278 DEBUG [Thread-1647] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:45,315 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=141 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/B/4ed251fa736c44c9b114969f480d818b 2023-04-12 15:54:45,378 DEBUG [Thread-1647] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:45,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:45,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:45,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:45,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:45,381 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:45,381 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:45,381 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:45,381 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:45,382 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:45,382 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:45,384 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,384 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:45,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:45,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:45,384 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,384 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:45,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:45,384 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:45,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:45,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:45,385 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:45,385 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:45,385 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:45,385 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:45,385 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:45,385 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:45,385 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:45,386 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:45,386 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:45,386 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. started... 2023-04-12 15:54:45,386 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. as already flushing 2023-04-12 15:54:45,386 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:45,386 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:45,386 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:45,386 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,386 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:45,386 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:45,387 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,387 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:45,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:45,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:45,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:45,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:45,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:45,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:45,389 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:45,389 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c3e703f[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:45,389 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:45,390 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:45,391 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,391 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:45,392 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:45,392 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:45,392 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:45,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:45,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:45,392 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:45,392 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:45,392 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:45,393 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:45,394 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:45,393 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,394 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:45,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:45,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:45,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:45,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:45,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:45,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:45,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:45,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:45,397 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:45,397 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:45,397 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:45,397 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:45,397 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:45,397 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:45,401 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:45,401 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:45,401 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:45,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:45,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:45,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:45,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:45,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:45,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:45,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:45,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:45,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:45,402 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:45,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:45,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:45,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:45,403 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:45,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,410 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:45,410 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:45,410 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:45,410 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:45,410 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:45,410 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:45,410 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:45,410 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:45,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:45,411 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,411 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:45,411 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:45,411 DEBUG [Thread-1647] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:45,411 DEBUG [Thread-1647] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:45,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:45,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:45,411 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:45,509 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:45,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53032 deadline: 1681314945507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,509 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:45,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53014 deadline: 1681314945507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,509 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:45,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53078 deadline: 1681314945507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,510 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:45,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53012 deadline: 1681314945509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,511 DEBUG [Thread-1647] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:45,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:45,513 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:45,514 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:45,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53040 deadline: 1681314945511, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:45,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:45,514 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:45,514 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:45,514 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:45,514 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:45,514 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:45,514 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:45,516 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,516 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:45,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:45,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:45,516 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,516 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:45,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:45,517 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:45,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:45,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:45,517 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:45,517 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:45,517 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:45,517 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:45,517 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:45,517 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:45,518 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:45,518 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:45,518 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:45,518 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. started... 2023-04-12 15:54:45,518 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. as already flushing 2023-04-12 15:54:45,518 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:45,518 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:45,518 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:45,518 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,518 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:45,518 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:45,519 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,519 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:45,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:45,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:45,520 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:45,520 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:45,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:45,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:45,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:45,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:45,521 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:45,521 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1075721[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:45,521 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:45,521 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:45,523 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:45,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:45,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:45,523 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:45,523 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:45,523 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:45,524 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,524 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:45,525 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:45,525 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:45,525 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,525 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:45,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:45,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:45,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:45,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:45,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:45,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:45,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:45,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:45,527 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:45,527 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:45,527 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:45,527 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:45,528 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:45,528 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:45,529 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:45,529 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:45,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:45,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:45,529 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:45,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:45,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:45,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:45,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:45,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:45,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:45,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:45,531 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:45,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:45,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:45,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,532 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:45,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:45,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,534 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,534 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:45,534 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:45,534 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:45,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:45,534 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:45,534 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:45,535 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:45,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:45,535 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:45,535 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,535 DEBUG [Thread-1647] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:45,535 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:45,535 DEBUG [Thread-1647] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:45,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:45,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:45,535 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:45,535 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:45,635 DEBUG [Thread-1647] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:45,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:45,637 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:45,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:45,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:45,638 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:45,639 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:45,639 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:45,639 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:45,639 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:45,639 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:45,641 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:45,641 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:45,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:45,642 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,642 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:45,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:45,642 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:45,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:45,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:45,642 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:45,643 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:45,643 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:45,643 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:45,643 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:45,643 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:45,643 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:45,643 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:45,643 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:45,643 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. started... 2023-04-12 15:54:45,643 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. as already flushing 2023-04-12 15:54:45,643 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:45,644 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:45,644 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:45,644 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,644 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:45,644 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:45,645 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,645 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:45,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:45,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:45,646 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:45,646 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:45,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:45,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:45,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:45,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:45,647 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:45,647 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76aa4bd2[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:45,647 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:45,647 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:45,648 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:45,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:45,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:45,648 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,649 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:45,649 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:45,649 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:45,649 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:45,650 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,650 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:45,650 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:45,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:45,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:45,650 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:45,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:45,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:45,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:45,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:45,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:45,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:45,652 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:45,652 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:45,652 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:45,652 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:45,652 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:45,652 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:45,655 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:45,655 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:45,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:45,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:45,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:45,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:45,655 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:45,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:45,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:45,656 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:45,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:45,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:45,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:45,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:45,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:45,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,658 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:45,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:45,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,660 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:45,660 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:45,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:45,660 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:45,660 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:45,660 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:45,660 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:45,660 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,660 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:45,660 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:45,660 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:45,660 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:45,660 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:45,660 DEBUG [Thread-1647] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:45,661 DEBUG [Thread-1647] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:45,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:45,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:45,725 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=141 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/C/b92af03fc80642d1bc59464f856b851a 2023-04-12 15:54:45,729 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/A/212f2c600b0a45ae8c36202a72f0fc40 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/212f2c600b0a45ae8c36202a72f0fc40 2023-04-12 15:54:45,734 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/212f2c600b0a45ae8c36202a72f0fc40, entries=250, sequenceid=141, filesize=16.4 K 2023-04-12 15:54:45,735 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/B/4ed251fa736c44c9b114969f480d818b as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/4ed251fa736c44c9b114969f480d818b 2023-04-12 15:54:45,738 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/4ed251fa736c44c9b114969f480d818b, entries=150, sequenceid=141, filesize=11.8 K 2023-04-12 15:54:45,739 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/C/b92af03fc80642d1bc59464f856b851a as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/b92af03fc80642d1bc59464f856b851a 2023-04-12 15:54:45,749 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/b92af03fc80642d1bc59464f856b851a, entries=150, sequenceid=141, filesize=11.8 K 2023-04-12 15:54:45,750 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=46.96 KB/48090 for deda1ce51fef7ef8d26da255318c55e4 in 868ms, sequenceid=141, compaction requested=true 2023-04-12 15:54:45,750 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for deda1ce51fef7ef8d26da255318c55e4: 2023-04-12 15:54:45,750 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:54:45,750 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:54:45,750 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-12 15:54:45,750 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-12 15:54:45,750 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:54:45,753 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 76246 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-12 15:54:45,754 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1912): deda1ce51fef7ef8d26da255318c55e4/A is initiating minor compaction (all files) 2023-04-12 15:54:45,754 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of deda1ce51fef7ef8d26da255318c55e4/A in TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:45,754 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/b99771cc50114253849cbfb2756ffd83, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/386394a2b715447892aee082948007ac, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/9ac3c8a644cc4147bdd1e1a75edd0021, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/05aa95731e504028a8a31f024a0756e3, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/45de1a82700642a5b0231e12f3276cb9, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/212f2c600b0a45ae8c36202a72f0fc40] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp, totalSize=74.5 K 2023-04-12 15:54:45,754 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting b99771cc50114253849cbfb2756ffd83, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=11, earliestPutTs=1681314879697 2023-04-12 15:54:45,754 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71466 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-12 15:54:45,754 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1912): deda1ce51fef7ef8d26da255318c55e4/B is initiating minor compaction (all files) 2023-04-12 15:54:45,755 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2259): Starting compaction of deda1ce51fef7ef8d26da255318c55e4/B in TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:45,755 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/a2523d426d53476cae46976432ba478f, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/c868446dfeb749948cabcda3c1f9c271, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/690c69166d0c44b2ba302fa0c8dce93f, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/c01ff948b9e242bb828790d2f8b52ac5, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/869399d589004bb1bec558eee7c12205, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/4ed251fa736c44c9b114969f480d818b] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp, totalSize=69.8 K 2023-04-12 15:54:45,755 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 386394a2b715447892aee082948007ac, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1681314879733 2023-04-12 15:54:45,755 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting a2523d426d53476cae46976432ba478f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=11, earliestPutTs=1681314879697 2023-04-12 15:54:45,756 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 9ac3c8a644cc4147bdd1e1a75edd0021, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=57, earliestPutTs=1681314880852 2023-04-12 15:54:45,756 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting c868446dfeb749948cabcda3c1f9c271, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1681314879733 2023-04-12 15:54:45,756 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 05aa95731e504028a8a31f024a0756e3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=93, earliestPutTs=1681314881997 2023-04-12 15:54:45,756 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 690c69166d0c44b2ba302fa0c8dce93f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=57, earliestPutTs=1681314880852 2023-04-12 15:54:45,757 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 45de1a82700642a5b0231e12f3276cb9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=104, earliestPutTs=1681314883120 2023-04-12 15:54:45,757 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting c01ff948b9e242bb828790d2f8b52ac5, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=93, earliestPutTs=1681314881997 2023-04-12 15:54:45,757 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 212f2c600b0a45ae8c36202a72f0fc40, keycount=250, bloomtype=ROW, size=16.4 K, encoding=NONE, compression=NONE, seqNum=141, earliestPutTs=1681314883765 2023-04-12 15:54:45,757 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 869399d589004bb1bec558eee7c12205, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=104, earliestPutTs=1681314883120 2023-04-12 15:54:45,758 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 4ed251fa736c44c9b114969f480d818b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=141, earliestPutTs=1681314883766 2023-04-12 15:54:45,761 DEBUG [Thread-1647] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:45,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:45,763 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:45,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:45,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:45,765 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:45,765 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:45,765 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:45,765 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:45,766 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:45,766 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:45,767 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,767 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:45,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:45,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:45,772 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:45,772 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:45,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:45,773 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:45,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:45,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:45,773 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:45,773 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:45,773 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:45,773 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:45,773 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:45,774 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:45,776 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:45,776 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:45,776 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:45,776 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. started... 2023-04-12 15:54:45,776 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing deda1ce51fef7ef8d26da255318c55e4 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-12 15:54:45,776 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK deda1ce51fef7ef8d26da255318c55e4, store=A 2023-04-12 15:54:45,776 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:45,776 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK deda1ce51fef7ef8d26da255318c55e4, store=B 2023-04-12 15:54:45,777 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:45,777 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK deda1ce51fef7ef8d26da255318c55e4, store=C 2023-04-12 15:54:45,777 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:45,782 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] throttle.PressureAwareThroughputController(145): deda1ce51fef7ef8d26da255318c55e4#A#compaction#219 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:54:45,785 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] throttle.PressureAwareThroughputController(145): deda1ce51fef7ef8d26da255318c55e4#B#compaction#220 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:54:46,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on deda1ce51fef7ef8d26da255318c55e4 2023-04-12 15:54:46,012 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. as already flushing 2023-04-12 15:54:46,030 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:46,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53012 deadline: 1681314946026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:46,031 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681314886014, max timestamp=1681314886029Num uniques 859; 2023-04-12 15:54:46,031 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681314886014, max timestamp=1681314886029Num uniques 859; 2023-04-12 15:54:46,031 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-12 15:54:46,031 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681314886014, max timestamp=1681314886029Num uniques 859; 2023-04-12 15:54:46,031 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-12 15:54:46,031 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-12 15:54:46,048 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:46,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53014 deadline: 1681314946044, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:46,048 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:46,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53078 deadline: 1681314946045, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:46,049 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:46,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53040 deadline: 1681314946045, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:46,049 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:46,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53032 deadline: 1681314946047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:46,133 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:46,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53012 deadline: 1681314946131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:46,151 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:46,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53014 deadline: 1681314946149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:46,152 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:46,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53078 deadline: 1681314946149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:46,152 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:46,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53040 deadline: 1681314946149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:46,152 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:46,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53032 deadline: 1681314946150, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:46,214 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/A/4073ebac96454afe80a6812184fff87b as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/4073ebac96454afe80a6812184fff87b 2023-04-12 15:54:46,219 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in deda1ce51fef7ef8d26da255318c55e4/A of deda1ce51fef7ef8d26da255318c55e4 into 4073ebac96454afe80a6812184fff87b(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:54:46,219 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for deda1ce51fef7ef8d26da255318c55e4: 2023-04-12 15:54:46,220 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4., storeName=deda1ce51fef7ef8d26da255318c55e4/A, priority=10, startTime=1681314885750; duration=0sec 2023-04-12 15:54:46,220 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:54:46,220 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-12 15:54:46,221 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/B/78ec40d786a84b8abae595ab823c553e as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/78ec40d786a84b8abae595ab823c553e 2023-04-12 15:54:46,222 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71466 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-12 15:54:46,222 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1912): deda1ce51fef7ef8d26da255318c55e4/C is initiating minor compaction (all files) 2023-04-12 15:54:46,222 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of deda1ce51fef7ef8d26da255318c55e4/C in TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:46,222 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/f00576ffc88f428cb64b448151eb504b, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/26c610ee0ad54ceb809f277f972104d4, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/9e703618cbd94eeda86c14a7ff44ae98, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/b2ab52c67a604d3ca27b2f62d5084af9, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/1f8ae434e8024e1cadd722956b5368bd, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/b92af03fc80642d1bc59464f856b851a] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp, totalSize=69.8 K 2023-04-12 15:54:46,222 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting f00576ffc88f428cb64b448151eb504b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=11, earliestPutTs=1681314879697 2023-04-12 15:54:46,223 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 26c610ee0ad54ceb809f277f972104d4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1681314879733 2023-04-12 15:54:46,223 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 9e703618cbd94eeda86c14a7ff44ae98, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=57, earliestPutTs=1681314880852 2023-04-12 15:54:46,224 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting b2ab52c67a604d3ca27b2f62d5084af9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=93, earliestPutTs=1681314881997 2023-04-12 15:54:46,224 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 1f8ae434e8024e1cadd722956b5368bd, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=104, earliestPutTs=1681314883120 2023-04-12 15:54:46,224 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting b92af03fc80642d1bc59464f856b851a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=141, earliestPutTs=1681314883766 2023-04-12 15:54:46,225 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in deda1ce51fef7ef8d26da255318c55e4/B of deda1ce51fef7ef8d26da255318c55e4 into 78ec40d786a84b8abae595ab823c553e(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:54:46,225 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for deda1ce51fef7ef8d26da255318c55e4: 2023-04-12 15:54:46,225 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4., storeName=deda1ce51fef7ef8d26da255318c55e4/B, priority=10, startTime=1681314885750; duration=0sec 2023-04-12 15:54:46,225 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:54:46,227 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=151 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/A/09fff26ac3c640a4aa119d81e2b362ca 2023-04-12 15:54:46,237 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] throttle.PressureAwareThroughputController(145): deda1ce51fef7ef8d26da255318c55e4#C#compaction#223 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:54:46,245 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/C/f1335b82a9124d7787242be2317595f3 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/f1335b82a9124d7787242be2317595f3 2023-04-12 15:54:46,249 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in deda1ce51fef7ef8d26da255318c55e4/C of deda1ce51fef7ef8d26da255318c55e4 into f1335b82a9124d7787242be2317595f3(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:54:46,250 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for deda1ce51fef7ef8d26da255318c55e4: 2023-04-12 15:54:46,250 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4., storeName=deda1ce51fef7ef8d26da255318c55e4/C, priority=10, startTime=1681314885750; duration=0sec 2023-04-12 15:54:46,250 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:54:46,337 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:46,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53012 deadline: 1681314946335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:46,353 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:46,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53014 deadline: 1681314946353, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:46,354 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:46,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53078 deadline: 1681314946353, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:46,354 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:46,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53040 deadline: 1681314946353, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:46,355 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:46,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53032 deadline: 1681314946354, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:46,639 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=151 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/B/c5f406c8d1da42aa88d83ecccb8d1335 2023-04-12 15:54:46,642 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:46,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53012 deadline: 1681314946641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:46,652 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=151 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/C/5c94082418524e11a8b3abeff6e0977b 2023-04-12 15:54:46,657 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:46,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53078 deadline: 1681314946656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:46,657 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:46,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53014 deadline: 1681314946656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:46,658 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:46,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53040 deadline: 1681314946656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:46,658 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:46,658 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/A/09fff26ac3c640a4aa119d81e2b362ca as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/09fff26ac3c640a4aa119d81e2b362ca 2023-04-12 15:54:46,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53032 deadline: 1681314946657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:46,664 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/09fff26ac3c640a4aa119d81e2b362ca, entries=150, sequenceid=151, filesize=11.8 K 2023-04-12 15:54:46,666 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/B/c5f406c8d1da42aa88d83ecccb8d1335 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/c5f406c8d1da42aa88d83ecccb8d1335 2023-04-12 15:54:46,670 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/c5f406c8d1da42aa88d83ecccb8d1335, entries=150, sequenceid=151, filesize=11.8 K 2023-04-12 15:54:46,675 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/C/5c94082418524e11a8b3abeff6e0977b as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/5c94082418524e11a8b3abeff6e0977b 2023-04-12 15:54:46,679 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/5c94082418524e11a8b3abeff6e0977b, entries=150, sequenceid=151, filesize=11.8 K 2023-04-12 15:54:46,680 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for deda1ce51fef7ef8d26da255318c55e4 in 904ms, sequenceid=151, compaction requested=false 2023-04-12 15:54:46,680 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for deda1ce51fef7ef8d26da255318c55e4: 2023-04-12 15:54:46,680 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:46,680 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:46,680 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:46,680 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:46,680 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:46,680 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:46,682 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:46,682 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:46,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:46,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:46,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:46,682 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:46,682 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:46,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:46,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:46,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:46,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:46,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:46,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:46,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35f152c4[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:46,685 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:46,685 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:46,687 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:46,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:46,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:46,687 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:46,687 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:46,687 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:46,687 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:46,687 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:46,688 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:46,688 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:46,688 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:46,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:46,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:46,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:46,688 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:46,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:46,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:46,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:46,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:46,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:46,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:46,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:46,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:46,691 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:46,691 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:46,691 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:46,691 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:46,691 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:46,691 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:46,693 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:46,693 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:46,693 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:46,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:46,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:46,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:46,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:46,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:46,694 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:46,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:46,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:46,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:46,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:46,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:46,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:46,695 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:46,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:46,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:46,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:46,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:46,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:46,701 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:46,701 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:46,701 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:46,701 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:46,701 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:46,701 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:46,701 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:46,702 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:46,701 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:46,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:46,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:46,702 DEBUG [Thread-1647] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:46,702 DEBUG [Thread-1647] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:46,701 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:46,702 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:46,702 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:46,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:46,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:46,703 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:46,802 DEBUG [Thread-1647] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:46,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:46,804 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:46,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:46,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:46,805 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:46,806 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:46,806 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:46,806 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:46,808 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:46,808 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:46,809 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:46,809 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:46,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:46,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:46,810 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:46,810 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:46,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:46,810 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:46,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:46,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:46,810 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:46,811 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:46,811 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:46,811 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:46,811 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:46,811 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:46,811 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:46,811 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:46,811 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:46,811 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. started... 2023-04-12 15:54:46,811 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing deda1ce51fef7ef8d26da255318c55e4 3/3 column families, dataSize=134.18 KB heapSize=375.61 KB 2023-04-12 15:54:46,812 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK deda1ce51fef7ef8d26da255318c55e4, store=A 2023-04-12 15:54:46,812 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:54:46,812 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK deda1ce51fef7ef8d26da255318c55e4, store=B 2023-04-12 15:54:46,812 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:54:46,812 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK deda1ce51fef7ef8d26da255318c55e4, store=C 2023-04-12 15:54:46,812 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:54:46,821 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=189 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/A/0152f039c6014feda7c02c0d81ee5d65 2023-04-12 15:54:47,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on deda1ce51fef7ef8d26da255318c55e4 2023-04-12 15:54:47,146 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. as already flushing 2023-04-12 15:54:47,170 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:47,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53078 deadline: 1681314947167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,171 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:47,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53040 deadline: 1681314947167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,171 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:47,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53032 deadline: 1681314947167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,171 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:47,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53012 deadline: 1681314947168, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,172 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:47,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53014 deadline: 1681314947170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,232 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=189 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/B/5d0fc1423a1e42bca92a105b87c6266e 2023-04-12 15:54:47,252 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=189 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/C/7dbb1a2863af44839f90798ab13b47f6 2023-04-12 15:54:47,258 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/A/0152f039c6014feda7c02c0d81ee5d65 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/0152f039c6014feda7c02c0d81ee5d65 2023-04-12 15:54:47,261 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/0152f039c6014feda7c02c0d81ee5d65, entries=150, sequenceid=189, filesize=11.8 K 2023-04-12 15:54:47,262 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/B/5d0fc1423a1e42bca92a105b87c6266e as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/5d0fc1423a1e42bca92a105b87c6266e 2023-04-12 15:54:47,266 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/5d0fc1423a1e42bca92a105b87c6266e, entries=150, sequenceid=189, filesize=11.8 K 2023-04-12 15:54:47,267 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/C/7dbb1a2863af44839f90798ab13b47f6 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/7dbb1a2863af44839f90798ab13b47f6 2023-04-12 15:54:47,271 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/7dbb1a2863af44839f90798ab13b47f6, entries=150, sequenceid=189, filesize=11.8 K 2023-04-12 15:54:47,271 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~375.56 KB/384576, currentSize=46.96 KB/48090 for deda1ce51fef7ef8d26da255318c55e4 in 460ms, sequenceid=189, compaction requested=true 2023-04-12 15:54:47,271 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for deda1ce51fef7ef8d26da255318c55e4: 2023-04-12 15:54:47,271 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:47,272 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:47,272 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:47,272 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,276 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:47,276 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:47,277 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,277 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:47,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:47,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:47,278 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:47,278 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:47,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:47,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:47,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:47,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:47,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:47,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a7163cd[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:47,280 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:47,280 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:47,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on deda1ce51fef7ef8d26da255318c55e4 2023-04-12 15:54:47,280 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing deda1ce51fef7ef8d26da255318c55e4 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-12 15:54:47,281 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:47,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:47,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:47,281 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:47,281 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:47,281 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:47,281 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,281 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:47,282 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK deda1ce51fef7ef8d26da255318c55e4, store=A 2023-04-12 15:54:47,282 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:47,282 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK deda1ce51fef7ef8d26da255318c55e4, store=B 2023-04-12 15:54:47,282 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:47,282 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK deda1ce51fef7ef8d26da255318c55e4, store=C 2023-04-12 15:54:47,282 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:47,284 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,284 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:47,284 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:47,284 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:47,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:47,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:47,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:47,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:47,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:47,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:47,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:47,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:47,287 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:47,287 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:47,288 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:47,288 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:47,288 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:47,288 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:47,290 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:47,290 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:47,290 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:47,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:47,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:47,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:47,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:47,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:47,290 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:47,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:47,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:47,292 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:47,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:47,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:47,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:47,294 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,294 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:47,295 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:47,295 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,295 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:47,295 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:47,295 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:47,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:47,295 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:47,295 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:47,295 DEBUG [Thread-1647] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:47,295 DEBUG [Thread-1647] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:47,295 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:47,295 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:47,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:47,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:47,295 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:47,296 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:47,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:47,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:47,302 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:47,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53014 deadline: 1681314947299, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,302 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:47,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53012 deadline: 1681314947299, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,303 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:47,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53032 deadline: 1681314947301, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,304 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:47,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53078 deadline: 1681314947302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,306 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=203 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/A/993f63e50c5846f69f90d9a04a376d3a 2023-04-12 15:54:47,395 DEBUG [Thread-1647] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:47,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:47,397 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:47,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:47,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:47,399 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:47,399 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:47,399 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:47,399 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:47,400 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:47,400 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:47,401 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:47,401 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:47,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:47,401 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,401 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:47,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:47,402 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:47,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:47,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:47,402 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:47,402 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:47,402 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:47,402 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:47,402 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:47,402 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:47,403 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:47,403 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:47,403 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:47,403 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. started... 2023-04-12 15:54:47,403 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. as already flushing 2023-04-12 15:54:47,403 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:47,403 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:47,403 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:47,403 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,403 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:47,403 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:47,404 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:47,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53014 deadline: 1681314947403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,405 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:47,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53012 deadline: 1681314947403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,406 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:47,406 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:47,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:47,406 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:47,406 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:47,406 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:47,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53032 deadline: 1681314947405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:47,407 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:47,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53078 deadline: 1681314947405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:47,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:47,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:47,408 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:47,408 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31f483fd[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:47,408 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:47,408 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:47,409 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:47,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:47,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:47,410 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:47,410 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:47,410 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:47,410 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,410 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:47,411 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:47,411 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,411 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:47,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:47,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:47,411 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:47,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:47,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:47,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:47,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:47,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:47,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:47,413 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:47,413 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:47,414 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:47,414 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:47,414 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:47,414 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:47,415 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:47,415 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:47,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:47,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:47,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:47,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:47,415 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:47,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:47,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:47,416 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:47,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:47,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:47,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:47,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:47,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:47,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,417 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:47,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:47,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,420 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,420 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:47,420 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:47,420 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,420 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:47,420 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:47,420 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:47,421 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:47,420 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:47,420 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:47,421 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:47,421 DEBUG [Thread-1647] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:47,421 DEBUG [Thread-1647] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:47,421 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:47,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:47,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:47,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:47,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:47,521 DEBUG [Thread-1647] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:47,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:47,523 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:47,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:47,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:47,524 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:47,524 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:47,524 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:47,524 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:47,524 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:47,525 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:47,526 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,526 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:47,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:47,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:47,526 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,526 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:47,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:47,526 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:47,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:47,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:47,527 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:47,527 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:47,527 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:47,527 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:47,528 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:47,528 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:47,528 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:47,528 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:47,528 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:47,528 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. started... 2023-04-12 15:54:47,528 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. as already flushing 2023-04-12 15:54:47,528 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:47,528 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:47,528 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:47,528 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,528 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:47,528 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:47,529 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:47,529 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:47,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:47,530 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:47,530 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:47,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:47,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:47,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:47,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:47,531 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:47,531 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1524aaca[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:47,531 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:47,531 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:47,532 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:47,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:47,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:47,532 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:47,532 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:47,532 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:47,532 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,532 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:47,534 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,534 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:47,534 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:47,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:47,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:47,534 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:47,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:47,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:47,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:47,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:47,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:47,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:47,537 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:47,537 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:47,537 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:47,537 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:47,537 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:47,537 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:47,538 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:47,539 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:47,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:47,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:47,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:47,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:47,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:47,539 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:47,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:47,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:47,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:47,539 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:47,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:47,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:47,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:47,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,540 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:47,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:47,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,543 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,543 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:47,543 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:47,543 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,543 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:47,543 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:47,543 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:47,543 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:47,543 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:47,543 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:47,543 DEBUG [Thread-1647] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:47,543 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:47,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:47,543 DEBUG [Thread-1647] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:47,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:47,543 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:47,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:47,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:47,608 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:47,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53014 deadline: 1681314947606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,608 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:47,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53012 deadline: 1681314947606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,609 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:47,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53078 deadline: 1681314947608, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,609 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:47,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53032 deadline: 1681314947608, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,643 DEBUG [Thread-1647] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:47,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:47,645 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:47,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:47,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:47,646 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:47,646 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:47,646 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:47,647 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:47,647 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:47,647 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:47,653 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:47,653 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:47,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:47,653 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,653 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:47,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:47,653 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:47,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:47,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:47,654 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:47,654 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:47,654 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:47,654 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:47,654 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:47,654 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:47,654 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:47,655 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:47,655 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:47,655 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. started... 2023-04-12 15:54:47,655 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. as already flushing 2023-04-12 15:54:47,655 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:47,655 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:47,655 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:47,655 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,655 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:47,655 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:47,659 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,659 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:47,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:47,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:47,660 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:47,660 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:47,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:47,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:47,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:47,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:47,661 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:47,661 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2afcbdd8[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:47,661 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:47,661 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:47,662 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:47,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:47,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:47,662 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:47,662 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:47,662 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:47,662 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,662 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:47,664 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:47,664 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,664 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:47,664 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:47,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:47,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:47,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:47,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:47,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:47,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:47,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:47,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:47,666 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:47,666 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:47,666 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:47,666 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:47,666 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:47,666 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:47,668 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:47,668 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:47,668 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:47,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:47,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:47,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:47,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:47,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:47,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:47,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:47,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:47,669 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:47,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:47,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:47,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:47,670 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:47,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:47,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,672 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:47,672 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,672 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:47,672 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:47,672 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:47,672 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:47,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:47,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:47,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:47,672 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,672 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:47,672 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:47,672 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:47,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:47,673 DEBUG [Thread-1647] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:47,673 DEBUG [Thread-1647] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:47,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:47,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:47,673 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:47,725 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=203 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/B/ee3a5f35d43045a6a41a8a2f4611b3ca 2023-04-12 15:54:47,773 DEBUG [Thread-1647] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:47,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:47,775 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:47,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:47,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:47,776 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:47,777 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:47,777 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:47,777 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:47,777 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:47,777 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:47,778 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,778 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:47,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:47,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:47,779 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,779 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:47,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:47,779 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:47,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:47,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:47,779 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:47,780 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:47,780 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:47,780 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:47,780 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:47,780 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:47,780 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:47,780 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:47,780 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:47,780 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. started... 2023-04-12 15:54:47,780 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. as already flushing 2023-04-12 15:54:47,780 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:47,780 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:47,780 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:47,780 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,780 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:47,780 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:47,782 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,782 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:47,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:47,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:47,782 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:47,782 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:47,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:47,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:47,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:47,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:47,784 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:47,784 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b12cc41[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:47,784 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:47,784 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:47,786 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:47,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:47,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:47,786 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:47,786 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:47,786 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:47,786 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,786 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:47,787 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:47,787 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:47,787 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:47,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:47,787 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:47,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:47,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:47,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:47,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:47,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:47,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:47,790 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:47,790 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:47,790 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:47,790 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:47,790 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:47,790 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:47,791 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:47,791 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:47,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:47,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:47,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:47,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:47,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:47,791 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:47,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:47,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:47,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:47,792 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:47,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:47,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:47,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:47,793 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:47,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:47,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,795 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:47,795 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:47,795 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:47,795 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:47,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:47,795 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:47,796 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:47,795 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:47,796 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:47,796 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:47,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:47,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:47,796 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,796 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:47,796 DEBUG [Thread-1647] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:47,796 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:47,796 DEBUG [Thread-1647] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:47,796 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:47,896 DEBUG [Thread-1647] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:47,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:47,898 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:47,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:47,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:47,900 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:47,900 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:47,900 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:47,900 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:47,900 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:47,900 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:47,903 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:47,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:47,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:47,903 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:47,903 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,903 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:47,904 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:47,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:47,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:47,904 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:47,904 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:47,904 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:47,904 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:47,904 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:47,904 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:47,904 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:47,904 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:47,904 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:47,904 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. started... 2023-04-12 15:54:47,905 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. as already flushing 2023-04-12 15:54:47,905 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:47,905 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:47,905 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:47,905 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,905 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:47,905 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:47,906 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,906 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:47,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:47,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:47,906 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:47,906 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:47,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:47,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:47,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:47,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:47,908 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:47,908 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e6c49b5[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:47,908 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:47,908 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:47,909 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:47,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:47,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:47,909 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,909 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:47,909 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:47,909 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:47,909 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:47,910 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:47,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:47,910 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:47,911 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:47,911 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:47,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:47,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:47,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:47,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:47,912 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:47,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53012 deadline: 1681314947910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,912 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:47,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:47,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53014 deadline: 1681314947911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:47,913 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:47,913 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:47,913 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:47,913 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:47,913 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:47,913 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:47,914 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:47,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53078 deadline: 1681314947912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,914 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:47,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53032 deadline: 1681314947912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,916 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:47,916 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:47,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:47,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:47,916 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:47,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:47,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:47,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:47,916 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:47,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:47,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:47,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:47,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:47,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:47,917 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:47,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:47,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:47,919 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,919 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:47,919 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:47,919 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:47,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:47,919 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:47,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:47,919 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:47,919 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:47,919 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:47,919 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:47,919 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:47,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:47,920 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:47,920 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:47,920 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:47,920 DEBUG [Thread-1647] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:47,920 DEBUG [Thread-1647] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:47,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:48,020 DEBUG [Thread-1647] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:48,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:48,022 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:48,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:48,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:48,023 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:48,023 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:48,023 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:48,023 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:48,024 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:48,024 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:48,025 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:48,025 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:48,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:48,025 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,025 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:48,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:48,025 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:48,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:48,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:48,026 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:48,026 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:48,026 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:48,026 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:48,026 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:48,026 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:48,026 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:48,026 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:48,026 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:48,026 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. started... 2023-04-12 15:54:48,027 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. as already flushing 2023-04-12 15:54:48,027 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:48,027 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:48,027 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:48,027 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,027 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:48,027 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:48,028 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,028 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:48,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:48,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:48,029 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:48,029 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:48,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:48,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:48,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:48,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:48,030 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:48,030 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e2ed648[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:48,030 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:48,030 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:48,032 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:48,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:48,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:48,032 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,032 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:48,032 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:48,032 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:48,032 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:48,033 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:48,033 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:48,033 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,033 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:48,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:48,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:48,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:48,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:48,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:48,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:48,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:48,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:48,036 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:48,036 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:48,036 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:48,036 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:48,036 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:48,036 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:48,037 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:48,037 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:48,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:48,037 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:48,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:48,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:48,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:48,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:48,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:48,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:48,038 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:48,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:48,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:48,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:48,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:48,039 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:48,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:48,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,041 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:48,041 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,041 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:48,041 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:48,041 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:48,041 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:48,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:48,041 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:48,042 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:48,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:48,042 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,042 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:48,042 DEBUG [Thread-1647] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:48,042 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:48,042 DEBUG [Thread-1647] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:48,042 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:48,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:48,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:48,136 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=203 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/C/2183b7bbeff84a27b052a71db45e6432 2023-04-12 15:54:48,141 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/A/993f63e50c5846f69f90d9a04a376d3a as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/993f63e50c5846f69f90d9a04a376d3a 2023-04-12 15:54:48,142 DEBUG [Thread-1647] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:48,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:48,144 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:48,145 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/993f63e50c5846f69f90d9a04a376d3a, entries=150, sequenceid=203, filesize=11.8 K 2023-04-12 15:54:48,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:48,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:48,145 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:48,145 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:48,146 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:48,146 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:48,146 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/B/ee3a5f35d43045a6a41a8a2f4611b3ca as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/ee3a5f35d43045a6a41a8a2f4611b3ca 2023-04-12 15:54:48,146 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:48,146 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:48,150 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/ee3a5f35d43045a6a41a8a2f4611b3ca, entries=150, sequenceid=203, filesize=11.8 K 2023-04-12 15:54:48,150 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/C/2183b7bbeff84a27b052a71db45e6432 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/2183b7bbeff84a27b052a71db45e6432 2023-04-12 15:54:48,151 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,151 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:48,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:48,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:48,151 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,151 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:48,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:48,152 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:48,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:48,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:48,153 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:48,153 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:48,153 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:48,153 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:48,153 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:48,153 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:48,153 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:48,153 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:48,153 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:48,153 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. started... 2023-04-12 15:54:48,154 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. as already flushing 2023-04-12 15:54:48,154 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:48,154 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:48,154 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:48,154 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,154 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:48,154 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:48,154 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/2183b7bbeff84a27b052a71db45e6432, entries=150, sequenceid=203, filesize=11.8 K 2023-04-12 15:54:48,155 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:48,155 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:48,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:48,155 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:48,155 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:48,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:48,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:48,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:48,156 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for deda1ce51fef7ef8d26da255318c55e4 in 876ms, sequenceid=203, compaction requested=true 2023-04-12 15:54:48,156 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for deda1ce51fef7ef8d26da255318c55e4: 2023-04-12 15:54:48,157 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-12 15:54:48,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,157 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:54:48,157 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-12 15:54:48,157 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:54:48,157 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:54:48,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:48,157 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:48,157 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4173a828[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:48,157 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:48,157 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:48,158 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48350 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-12 15:54:48,158 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48350 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-12 15:54:48,158 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1912): deda1ce51fef7ef8d26da255318c55e4/B is initiating minor compaction (all files) 2023-04-12 15:54:48,158 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2259): Starting compaction of deda1ce51fef7ef8d26da255318c55e4/B in TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:48,158 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:48,158 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1912): deda1ce51fef7ef8d26da255318c55e4/A is initiating minor compaction (all files) 2023-04-12 15:54:48,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:48,159 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of deda1ce51fef7ef8d26da255318c55e4/A in TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:48,158 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/78ec40d786a84b8abae595ab823c553e, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/c5f406c8d1da42aa88d83ecccb8d1335, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/5d0fc1423a1e42bca92a105b87c6266e, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/ee3a5f35d43045a6a41a8a2f4611b3ca] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp, totalSize=47.2 K 2023-04-12 15:54:48,159 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/4073ebac96454afe80a6812184fff87b, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/09fff26ac3c640a4aa119d81e2b362ca, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/0152f039c6014feda7c02c0d81ee5d65, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/993f63e50c5846f69f90d9a04a376d3a] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp, totalSize=47.2 K 2023-04-12 15:54:48,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:48,159 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:48,159 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:48,159 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:48,159 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,159 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:48,159 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 78ec40d786a84b8abae595ab823c553e, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=141, earliestPutTs=1681314883766 2023-04-12 15:54:48,159 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 4073ebac96454afe80a6812184fff87b, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=141, earliestPutTs=1681314883766 2023-04-12 15:54:48,160 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting c5f406c8d1da42aa88d83ecccb8d1335, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=151, earliestPutTs=1681314884884 2023-04-12 15:54:48,161 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,161 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:48,161 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:48,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:48,161 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 09fff26ac3c640a4aa119d81e2b362ca, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=151, earliestPutTs=1681314884884 2023-04-12 15:54:48,161 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:48,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:48,161 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 5d0fc1423a1e42bca92a105b87c6266e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=189, earliestPutTs=1681314886038 2023-04-12 15:54:48,162 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 0152f039c6014feda7c02c0d81ee5d65, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=189, earliestPutTs=1681314886038 2023-04-12 15:54:48,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:48,162 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting ee3a5f35d43045a6a41a8a2f4611b3ca, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1681314887279 2023-04-12 15:54:48,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:48,162 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 993f63e50c5846f69f90d9a04a376d3a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1681314887279 2023-04-12 15:54:48,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:48,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:48,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:48,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:48,165 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:48,165 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:48,165 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:48,165 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:48,165 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:48,165 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:48,168 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:48,168 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:48,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:48,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:48,168 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:48,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:48,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:48,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:48,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:48,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:48,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:48,169 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:48,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:48,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:48,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:48,172 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] throttle.PressureAwareThroughputController(145): deda1ce51fef7ef8d26da255318c55e4#B#compaction#231 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:54:48,175 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] throttle.PressureAwareThroughputController(145): deda1ce51fef7ef8d26da255318c55e4#A#compaction#232 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:54:48,176 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:48,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:48,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,187 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,187 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:48,187 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:48,187 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:48,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:48,187 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:48,187 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:48,187 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:48,187 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:48,187 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:48,187 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:48,187 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:48,187 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:48,187 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:48,187 DEBUG [Thread-1647] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:48,187 DEBUG [Thread-1647] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:48,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:48,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:48,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on deda1ce51fef7ef8d26da255318c55e4 2023-04-12 15:54:48,188 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing deda1ce51fef7ef8d26da255318c55e4 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-12 15:54:48,189 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK deda1ce51fef7ef8d26da255318c55e4, store=A 2023-04-12 15:54:48,189 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:48,189 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK deda1ce51fef7ef8d26da255318c55e4, store=B 2023-04-12 15:54:48,189 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:48,189 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK deda1ce51fef7ef8d26da255318c55e4, store=C 2023-04-12 15:54:48,189 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:48,197 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=222 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/A/7295be13e3ff4465866d021d82f31e76 2023-04-12 15:54:48,205 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/A/719ca553c87d49ae9ef9e7f4a0507870 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/719ca553c87d49ae9ef9e7f4a0507870 2023-04-12 15:54:48,212 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in deda1ce51fef7ef8d26da255318c55e4/A of deda1ce51fef7ef8d26da255318c55e4 into 719ca553c87d49ae9ef9e7f4a0507870(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:54:48,212 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for deda1ce51fef7ef8d26da255318c55e4: 2023-04-12 15:54:48,212 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4., storeName=deda1ce51fef7ef8d26da255318c55e4/A, priority=12, startTime=1681314888156; duration=0sec 2023-04-12 15:54:48,212 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:54:48,212 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-12 15:54:48,213 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48350 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-12 15:54:48,213 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1912): deda1ce51fef7ef8d26da255318c55e4/C is initiating minor compaction (all files) 2023-04-12 15:54:48,213 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of deda1ce51fef7ef8d26da255318c55e4/C in TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:48,213 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/f1335b82a9124d7787242be2317595f3, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/5c94082418524e11a8b3abeff6e0977b, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/7dbb1a2863af44839f90798ab13b47f6, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/2183b7bbeff84a27b052a71db45e6432] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp, totalSize=47.2 K 2023-04-12 15:54:48,214 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting f1335b82a9124d7787242be2317595f3, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=141, earliestPutTs=1681314883766 2023-04-12 15:54:48,214 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 5c94082418524e11a8b3abeff6e0977b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=151, earliestPutTs=1681314884884 2023-04-12 15:54:48,217 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 7dbb1a2863af44839f90798ab13b47f6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=189, earliestPutTs=1681314886038 2023-04-12 15:54:48,217 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 2183b7bbeff84a27b052a71db45e6432, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1681314887279 2023-04-12 15:54:48,223 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] throttle.PressureAwareThroughputController(145): deda1ce51fef7ef8d26da255318c55e4#C#compaction#235 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:54:48,231 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/C/754391c76d23499b9475cc8a247ace5e as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/754391c76d23499b9475cc8a247ace5e 2023-04-12 15:54:48,232 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:48,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53040 deadline: 1681314948230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,235 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in deda1ce51fef7ef8d26da255318c55e4/C of deda1ce51fef7ef8d26da255318c55e4 into 754391c76d23499b9475cc8a247ace5e(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:54:48,235 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for deda1ce51fef7ef8d26da255318c55e4: 2023-04-12 15:54:48,236 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4., storeName=deda1ce51fef7ef8d26da255318c55e4/C, priority=12, startTime=1681314888157; duration=0sec 2023-04-12 15:54:48,236 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:54:48,288 DEBUG [Thread-1647] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:48,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:48,290 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:48,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:48,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:48,291 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:48,291 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:48,291 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:48,291 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:48,291 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:48,291 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:48,292 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:48,292 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:48,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:48,292 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,292 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:48,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:48,293 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:48,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:48,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:48,293 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:48,293 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:48,293 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:48,293 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:48,293 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:48,294 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:48,294 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:48,294 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:48,294 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:48,294 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. started... 2023-04-12 15:54:48,294 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. as already flushing 2023-04-12 15:54:48,294 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:48,294 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:48,294 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:48,294 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,294 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:48,294 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:48,295 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,295 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:48,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:48,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:48,295 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:48,296 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:48,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:48,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:48,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:48,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:48,297 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:48,297 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7cbe0483[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:48,297 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:48,297 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:48,301 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:48,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:48,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:48,301 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:48,301 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:48,301 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,302 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:48,301 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:48,303 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,303 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:48,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:48,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:48,303 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:48,303 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:48,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:48,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:48,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:48,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:48,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:48,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:48,305 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:48,305 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:48,305 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:48,305 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:48,305 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:48,305 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:48,307 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:48,307 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:48,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:48,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:48,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:48,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:48,307 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:48,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:48,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:48,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:48,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:48,308 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:48,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:48,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:48,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:48,308 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:48,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:48,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,310 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,310 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:48,310 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:48,311 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:48,311 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:48,311 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:48,311 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,311 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:48,311 DEBUG [Thread-1647] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:48,311 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:48,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:48,311 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:48,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:48,311 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:48,311 DEBUG [Thread-1647] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:48,311 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:48,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:48,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:48,334 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:48,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53040 deadline: 1681314948333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,411 DEBUG [Thread-1647] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:48,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:48,413 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:48,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:48,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:48,414 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:48,414 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:48,415 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:48,415 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:48,415 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:48,415 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:48,416 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:48,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53014 deadline: 1681314948414, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,417 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:48,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53032 deadline: 1681314948416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,418 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:48,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53012 deadline: 1681314948416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,418 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:48,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53078 deadline: 1681314948417, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,419 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:48,419 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:48,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:48,419 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,419 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:48,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:48,419 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:48,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:48,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:48,420 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:48,420 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:48,420 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:48,420 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:48,420 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:48,420 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:48,420 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:48,420 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:48,420 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:48,421 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. started... 2023-04-12 15:54:48,421 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. as already flushing 2023-04-12 15:54:48,421 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:48,421 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:48,421 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:48,421 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,421 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:48,421 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:48,422 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:48,422 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:48,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:48,423 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:48,423 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:48,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:48,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:48,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:48,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:48,424 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:48,424 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2043cb40[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:48,424 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:48,424 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:48,426 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:48,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:48,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:48,426 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:48,426 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:48,426 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:48,426 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,426 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:48,431 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:48,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:48,431 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:48,431 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:48,432 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:48,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:48,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:48,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:48,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:48,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:48,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:48,434 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:48,434 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:48,434 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:48,434 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:48,434 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:48,434 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:48,435 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:48,435 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:48,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:48,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:48,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:48,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:48,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:48,435 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:48,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:48,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:48,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:48,436 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:48,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:48,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:48,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:48,437 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:48,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:48,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,439 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,439 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:48,439 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:48,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:48,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:48,439 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:48,439 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:48,439 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:48,439 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:48,439 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:48,440 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:48,440 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,440 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:48,440 DEBUG [Thread-1647] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:48,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:48,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:48,440 DEBUG [Thread-1647] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:48,440 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:48,440 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:48,539 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:48,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53040 deadline: 1681314948536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,540 DEBUG [Thread-1647] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:48,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:48,542 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:48,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:48,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:48,543 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:48,543 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:48,543 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:48,543 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:48,543 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:48,543 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:48,544 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,544 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:48,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:48,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:48,545 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,545 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:48,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:48,545 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:48,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:48,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:48,546 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:48,546 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:48,546 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:48,546 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:48,546 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:48,546 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:48,546 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:48,546 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:48,546 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:48,546 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. started... 2023-04-12 15:54:48,546 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. as already flushing 2023-04-12 15:54:48,546 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:48,546 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:48,546 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:48,546 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,546 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:48,546 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:48,549 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:48,549 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:48,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:48,549 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:48,549 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:48,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:48,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:48,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:48,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:48,550 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:48,550 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5827d596[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:48,550 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:48,551 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:48,552 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:48,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:48,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:48,552 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:48,552 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:48,552 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:48,552 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,552 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:48,553 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,553 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:48,553 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:48,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:48,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:48,553 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:48,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:48,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:48,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:48,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:48,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:48,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:48,555 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:48,555 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:48,555 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:48,555 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:48,555 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:48,555 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:48,557 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:48,557 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:48,557 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:48,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:48,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:48,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:48,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:48,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:48,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:48,557 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:48,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:48,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:48,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:48,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:48,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:48,559 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:48,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:48,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,562 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:48,562 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,562 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:48,562 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:48,562 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:48,562 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:48,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:48,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:48,562 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:48,562 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:48,562 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,562 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:48,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:48,562 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:48,562 DEBUG [Thread-1647] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:48,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:48,562 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:48,562 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:48,562 DEBUG [Thread-1647] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:48,591 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/B/9a066fa7382641d1ac5d8d9609b9ca03 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/9a066fa7382641d1ac5d8d9609b9ca03 2023-04-12 15:54:48,596 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in deda1ce51fef7ef8d26da255318c55e4/B of deda1ce51fef7ef8d26da255318c55e4 into 9a066fa7382641d1ac5d8d9609b9ca03(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:54:48,596 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for deda1ce51fef7ef8d26da255318c55e4: 2023-04-12 15:54:48,596 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4., storeName=deda1ce51fef7ef8d26da255318c55e4/B, priority=12, startTime=1681314888157; duration=0sec 2023-04-12 15:54:48,596 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:54:48,612 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=222 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/B/27cc961916fc42658d3d2f3d0be19021 2023-04-12 15:54:48,663 DEBUG [Thread-1647] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:48,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:48,665 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:48,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:48,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:48,666 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:48,666 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:48,666 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:48,666 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:48,666 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:48,666 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:48,668 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:48,668 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:48,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:48,668 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,668 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:48,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:48,669 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:48,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:48,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:48,669 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:48,669 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:48,670 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:48,670 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:48,670 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:48,670 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:48,670 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:48,670 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:48,670 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:48,670 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. started... 2023-04-12 15:54:48,670 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. as already flushing 2023-04-12 15:54:48,670 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:48,670 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:48,670 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:48,670 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,670 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:48,670 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:48,672 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:48,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:48,672 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:48,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:48,672 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:48,673 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:48,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:48,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:48,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:48,674 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:48,674 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:48,674 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4490452[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:48,674 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:48,675 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:48,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:48,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:48,675 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:48,675 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:48,675 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,675 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:48,675 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:48,678 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:48,678 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:48,678 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,678 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:48,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:48,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:48,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:48,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:48,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:48,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:48,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:48,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:48,681 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:48,682 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:48,682 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:48,682 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:48,682 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:48,682 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:48,683 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:48,683 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:48,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:48,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:48,683 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:48,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:48,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:48,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:48,684 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:48,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:48,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:48,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:48,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:48,685 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:48,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:48,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:48,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:48,687 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,687 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:48,687 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:48,687 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:48,687 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:48,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:48,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:48,687 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:48,687 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:48,688 DEBUG [Thread-1647] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:48,687 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:48,688 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,688 DEBUG [Thread-1647] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:48,688 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:48,688 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:48,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:48,688 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:48,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:48,688 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:48,788 DEBUG [Thread-1647] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:48,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:48,790 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:48,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:48,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:48,791 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:48,791 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:48,791 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:48,791 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:48,792 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:48,792 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:48,793 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:48,793 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:48,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:48,793 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,793 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:48,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:48,794 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:48,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:48,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:48,794 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:48,794 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:48,794 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:48,794 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:48,794 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:48,795 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:48,795 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:48,795 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:48,795 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:48,795 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. started... 2023-04-12 15:54:48,795 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. as already flushing 2023-04-12 15:54:48,795 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:48,795 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:48,795 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:48,795 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,795 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:48,795 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:48,797 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,797 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:48,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:48,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:48,798 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:48,798 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:48,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:48,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:48,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:48,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:48,800 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:48,800 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@85f8244[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:48,800 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:48,800 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:48,801 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:48,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:48,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:48,801 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:48,801 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:48,801 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:48,801 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,801 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:48,802 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,802 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:48,802 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:48,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:48,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:48,802 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:48,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:48,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:48,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:48,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:48,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:48,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:48,805 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:48,805 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:48,805 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:48,805 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:48,805 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:48,805 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:48,807 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:48,807 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:48,807 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:48,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:48,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:48,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:48,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:48,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:48,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:48,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:48,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:48,807 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:48,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:48,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:48,809 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:48,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:48,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:48,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,814 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:48,814 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:48,814 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:48,814 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:48,814 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:48,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:48,814 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:48,814 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:48,814 DEBUG [Thread-1647] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:48,814 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:48,814 DEBUG [Thread-1647] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:48,815 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,815 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:48,815 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:48,815 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:48,815 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:48,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:48,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:48,841 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:48,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53040 deadline: 1681314948840, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,915 DEBUG [Thread-1647] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:48,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:48,917 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:48,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:48,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:48,918 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:48,918 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:48,918 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:48,918 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:48,918 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:48,919 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:48,920 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,920 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:48,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:48,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:48,920 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,920 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:48,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:48,920 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:48,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:48,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:48,921 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:48,921 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:48,921 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:48,921 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:48,921 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:48,921 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:48,921 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:48,921 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:48,921 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:48,921 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. started... 2023-04-12 15:54:48,922 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. as already flushing 2023-04-12 15:54:48,922 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:48,922 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:48,922 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:48,922 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,922 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:48,922 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:48,923 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,923 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:48,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:48,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:48,923 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:48,923 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:48,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:48,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:48,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:48,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:48,925 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:48,925 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36f841c4[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:48,925 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:48,925 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:48,926 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:48,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:48,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:48,926 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:48,926 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:48,926 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:48,926 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,926 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:48,928 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,928 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:48,928 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:48,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:48,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:48,928 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:48,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:48,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:48,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:48,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:48,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:48,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:48,931 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:48,931 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:48,931 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:48,931 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:48,931 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:48,931 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:48,932 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:48,932 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:48,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:48,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:48,932 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:48,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:48,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:48,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:48,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:48,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:48,933 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:48,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:48,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:48,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:48,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:48,934 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:48,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:48,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,937 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,937 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:48,937 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:48,937 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:48,937 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:48,937 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:48,937 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:48,937 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:48,937 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:48,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:48,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:48,937 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:48,937 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:48,938 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:48,938 DEBUG [Thread-1647] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:48,938 DEBUG [Thread-1647] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:48,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:48,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:49,032 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=222 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/C/d43f8ae378264d50b474ef7c7fbf544e 2023-04-12 15:54:49,036 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/A/7295be13e3ff4465866d021d82f31e76 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/7295be13e3ff4465866d021d82f31e76 2023-04-12 15:54:49,038 DEBUG [Thread-1647] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:49,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:49,040 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:49,040 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/7295be13e3ff4465866d021d82f31e76, entries=200, sequenceid=222, filesize=14.1 K 2023-04-12 15:54:49,044 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/B/27cc961916fc42658d3d2f3d0be19021 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/27cc961916fc42658d3d2f3d0be19021 2023-04-12 15:54:49,048 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/27cc961916fc42658d3d2f3d0be19021, entries=150, sequenceid=222, filesize=11.8 K 2023-04-12 15:54:49,049 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/C/d43f8ae378264d50b474ef7c7fbf544e as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/d43f8ae378264d50b474ef7c7fbf544e 2023-04-12 15:54:49,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:49,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:49,052 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:49,052 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:49,052 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:49,052 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:49,052 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:49,052 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:49,054 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/d43f8ae378264d50b474ef7c7fbf544e, entries=150, sequenceid=222, filesize=11.8 K 2023-04-12 15:54:49,054 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:49,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:49,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:49,054 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,055 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for deda1ce51fef7ef8d26da255318c55e4 in 867ms, sequenceid=222, compaction requested=false 2023-04-12 15:54:49,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:49,055 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for deda1ce51fef7ef8d26da255318c55e4: 2023-04-12 15:54:49,055 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,055 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:49,055 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:49,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:49,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:49,055 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:49,056 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:49,056 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:49,056 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:49,056 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:49,056 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:49,056 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:49,056 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:49,056 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:49,057 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. started... 2023-04-12 15:54:49,057 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing deda1ce51fef7ef8d26da255318c55e4 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-12 15:54:49,057 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK deda1ce51fef7ef8d26da255318c55e4, store=A 2023-04-12 15:54:49,057 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:49,057 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK deda1ce51fef7ef8d26da255318c55e4, store=B 2023-04-12 15:54:49,057 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:49,057 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK deda1ce51fef7ef8d26da255318c55e4, store=C 2023-04-12 15:54:49,057 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:49,081 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=238 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/A/b654ff237113487d8e5a691b7c9c0d94 2023-04-12 15:54:49,093 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=238 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/B/cf1523cb7d924a47b293492713c6b0cc 2023-04-12 15:54:49,103 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=238 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/C/88bd2a2da3b040df9ea5948d729d9742 2023-04-12 15:54:49,107 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/A/b654ff237113487d8e5a691b7c9c0d94 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/b654ff237113487d8e5a691b7c9c0d94 2023-04-12 15:54:49,110 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/b654ff237113487d8e5a691b7c9c0d94, entries=150, sequenceid=238, filesize=11.8 K 2023-04-12 15:54:49,111 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/B/cf1523cb7d924a47b293492713c6b0cc as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/cf1523cb7d924a47b293492713c6b0cc 2023-04-12 15:54:49,117 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/cf1523cb7d924a47b293492713c6b0cc, entries=150, sequenceid=238, filesize=11.8 K 2023-04-12 15:54:49,118 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/C/88bd2a2da3b040df9ea5948d729d9742 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/88bd2a2da3b040df9ea5948d729d9742 2023-04-12 15:54:49,125 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/88bd2a2da3b040df9ea5948d729d9742, entries=150, sequenceid=238, filesize=11.8 K 2023-04-12 15:54:49,126 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=0 B/0 for deda1ce51fef7ef8d26da255318c55e4 in 69ms, sequenceid=238, compaction requested=true 2023-04-12 15:54:49,126 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for deda1ce51fef7ef8d26da255318c55e4: 2023-04-12 15:54:49,126 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:49,126 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:49,126 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:49,126 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,128 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:49,128 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:49,129 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,129 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:49,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:49,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:49,130 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:49,130 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:49,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:49,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:49,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:49,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:49,132 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:49,132 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:49,132 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5015a9e8[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:49,132 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:49,133 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,133 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:49,134 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:49,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:49,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:49,134 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:49,134 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:49,134 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:49,137 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,137 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:49,137 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:49,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:49,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:49,137 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:49,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:49,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:49,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:49,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:49,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:49,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:49,139 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:49,139 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:49,139 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:49,139 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:49,140 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:49,140 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:49,141 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:49,141 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:49,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:49,141 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:49,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:49,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:49,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:49,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:49,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:49,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:49,142 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:49,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:49,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:49,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:49,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:49,145 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:49,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:49,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,147 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,147 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:49,147 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:49,147 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:49,147 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:49,147 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:49,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:49,147 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:49,148 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:49,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:49,147 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,148 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:49,148 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:49,148 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:49,148 DEBUG [Thread-1647] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:49,148 DEBUG [Thread-1647] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:49,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:49,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:49,148 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:49,248 DEBUG [Thread-1647] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:49,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:49,250 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:49,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:49,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:49,251 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:49,251 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:49,251 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:49,251 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:49,252 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:49,252 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:49,253 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:49,253 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:49,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:49,253 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,254 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:49,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:49,254 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:49,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:49,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:49,254 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:49,255 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:49,255 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:49,255 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:49,255 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:49,255 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:49,255 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:49,255 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:49,255 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:49,255 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. started... 2023-04-12 15:54:49,255 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for deda1ce51fef7ef8d26da255318c55e4: 2023-04-12 15:54:49,255 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:49,255 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:49,255 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:49,255 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,256 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:49,256 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:49,258 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,258 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:49,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:49,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:49,259 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:49,259 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:49,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:49,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:49,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:49,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:49,260 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:49,260 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c325c2c[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:49,260 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:49,260 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:49,261 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:49,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:49,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:49,261 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,261 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:49,261 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:49,261 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:49,261 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:49,263 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:49,263 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:49,263 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,263 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:49,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:49,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:49,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:49,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:49,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:49,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:49,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:49,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:49,265 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:49,265 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:49,265 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:49,265 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:49,265 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:49,265 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:49,266 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:49,266 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:49,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:49,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:49,266 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:49,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:49,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:49,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:49,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:49,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:49,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:49,267 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:49,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:49,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:49,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:49,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,268 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:49,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:49,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,275 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,275 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:49,275 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:49,275 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:49,275 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:49,275 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:49,275 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:49,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:49,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:49,275 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:49,275 DEBUG [Thread-1647] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:49,275 DEBUG [Thread-1647] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:49,275 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,275 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:49,275 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:49,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:49,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:49,275 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:49,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on deda1ce51fef7ef8d26da255318c55e4 2023-04-12 15:54:49,359 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing deda1ce51fef7ef8d26da255318c55e4 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-12 15:54:49,359 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK deda1ce51fef7ef8d26da255318c55e4, store=A 2023-04-12 15:54:49,359 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:49,359 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK deda1ce51fef7ef8d26da255318c55e4, store=B 2023-04-12 15:54:49,359 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:49,359 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK deda1ce51fef7ef8d26da255318c55e4, store=C 2023-04-12 15:54:49,359 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:49,375 DEBUG [Thread-1647] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:49,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:49,377 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:49,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:49,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:49,378 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:49,378 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:49,378 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:49,378 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:49,379 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:49,379 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:49,380 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:49,380 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:49,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:49,380 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,380 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:49,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:49,381 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:49,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:49,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:49,381 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:49,381 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:49,381 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:49,381 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:49,381 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:49,381 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:49,382 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:49,382 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:49,382 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:49,382 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. started... 2023-04-12 15:54:49,382 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. as already flushing 2023-04-12 15:54:49,382 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:49,382 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:49,382 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:49,382 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,382 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:49,382 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:49,383 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:49,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:49,383 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:49,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:49,384 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:49,384 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:49,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:49,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:49,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:49,386 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:49,386 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@680bb04c[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:49,386 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:49,386 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:49,387 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:49,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:49,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:49,387 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:49,387 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:49,387 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:49,388 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,388 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:49,390 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,390 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:49,390 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:49,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:49,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:49,390 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:49,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:49,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:49,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:49,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:49,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:49,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:49,393 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:49,393 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:49,393 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:49,393 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:49,393 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:49,393 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:49,395 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:49,395 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:49,395 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:49,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:49,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:49,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:49,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:49,396 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:49,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:49,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:49,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:49,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:49,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:49,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:49,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:49,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,403 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:49,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:49,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,405 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:49,405 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:49,405 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:49,405 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:49,405 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:49,405 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:49,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:49,405 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:49,405 DEBUG [Thread-1647] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:49,406 DEBUG [Thread-1647] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:49,405 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:49,406 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,406 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:49,406 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:49,406 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:49,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:49,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:49,426 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681314889396, max timestamp=1681314889424Num uniques 859; 2023-04-12 15:54:49,426 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-12 15:54:49,426 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681314889396, max timestamp=1681314889424Num uniques 859; 2023-04-12 15:54:49,426 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-12 15:54:49,426 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681314889396, max timestamp=1681314889424Num uniques 859; 2023-04-12 15:54:49,426 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-12 15:54:49,440 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:49,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53040 deadline: 1681314949437, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,443 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:49,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53032 deadline: 1681314949439, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,443 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:49,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53012 deadline: 1681314949440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,445 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:49,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53014 deadline: 1681314949441, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,446 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:49,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53078 deadline: 1681314949441, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,506 DEBUG [Thread-1647] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:49,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:49,508 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:49,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:49,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:49,509 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:49,509 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:49,509 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:49,509 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:49,510 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:49,510 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:49,511 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,511 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:49,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:49,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:49,511 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,511 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:49,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:49,511 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:49,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:49,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:49,512 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:49,512 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:49,512 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:49,512 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:49,512 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:49,512 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:49,512 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:49,512 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:49,512 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:49,513 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. started... 2023-04-12 15:54:49,513 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. as already flushing 2023-04-12 15:54:49,513 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:49,513 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:49,513 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:49,513 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,513 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:49,513 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:49,515 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:49,515 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:49,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:49,516 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:49,516 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:49,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:49,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:49,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:49,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:49,517 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:49,517 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c092ae8[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:49,517 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:49,517 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:49,518 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:49,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:49,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:49,519 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:49,519 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:49,519 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,519 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:49,519 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:49,520 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:49,520 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:49,520 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,520 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:49,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:49,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:49,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:49,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:49,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:49,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:49,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:49,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:49,523 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:49,523 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:49,523 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:49,523 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:49,523 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:49,523 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:49,524 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:49,524 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:49,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:49,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:49,524 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:49,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:49,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:49,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:49,525 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:49,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:49,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:49,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:49,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:49,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:49,526 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:49,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:49,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:49,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,533 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,533 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:49,533 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:49,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:49,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:49,533 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:49,533 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:49,533 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:49,533 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:49,533 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:49,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:49,533 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:49,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:49,533 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,533 DEBUG [Thread-1647] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:49,533 DEBUG [Thread-1647] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:49,533 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:49,533 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:49,533 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:49,544 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:49,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53040 deadline: 1681314949541, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,546 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:49,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53032 deadline: 1681314949544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,546 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:49,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53012 deadline: 1681314949544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,549 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:49,549 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:49,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53014 deadline: 1681314949546, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53078 deadline: 1681314949546, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,633 DEBUG [Thread-1647] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:49,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:49,635 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:49,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:49,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:49,636 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:49,636 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:49,636 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:49,636 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:49,637 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:49,637 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:49,638 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,638 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:49,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:49,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:49,638 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,638 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:49,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:49,639 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:49,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:49,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:49,639 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:49,639 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:49,639 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:49,639 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:49,639 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:49,639 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:49,639 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:49,639 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:49,639 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:49,639 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. started... 2023-04-12 15:54:49,640 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. as already flushing 2023-04-12 15:54:49,640 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:49,640 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:49,640 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:49,640 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,640 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:49,640 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:49,641 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:49,641 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:49,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:49,641 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:49,641 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:49,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:49,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:49,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:49,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:49,643 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:49,643 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61ae8a68[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:49,643 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:49,643 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:49,644 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:49,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:49,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:49,644 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:49,644 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:49,644 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,644 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:49,644 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:49,647 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:49,647 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:49,647 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,647 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:49,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:49,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:49,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:49,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:49,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:49,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:49,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:49,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:49,649 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:49,649 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:49,649 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:49,649 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:49,649 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:49,649 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:49,650 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:49,650 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:49,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:49,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:49,650 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:49,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:49,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:49,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:49,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:49,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:49,651 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:49,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:49,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:49,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:49,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:49,652 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:49,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:49,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,654 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,654 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:49,654 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:49,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:49,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:49,654 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:49,654 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:49,654 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:49,654 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:49,654 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:49,655 DEBUG [Thread-1647] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:49,654 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:49,655 DEBUG [Thread-1647] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:49,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:49,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:49,655 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,655 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:49,655 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:49,655 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:49,747 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:49,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53040 deadline: 1681314949745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,750 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:49,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53012 deadline: 1681314949747, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,750 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:49,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53032 deadline: 1681314949748, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,752 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:49,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53014 deadline: 1681314949750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,753 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:49,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53078 deadline: 1681314949751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,755 DEBUG [Thread-1647] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:49,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:49,757 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:49,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:49,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:49,758 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:49,758 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:49,758 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:49,758 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:49,758 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:49,758 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:49,763 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:49,763 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:49,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:49,763 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:49,763 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:49,763 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:49,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:49,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:49,764 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:49,764 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:49,764 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:49,764 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:49,764 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:49,764 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:49,764 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:49,764 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:49,764 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:49,765 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. started... 2023-04-12 15:54:49,765 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. as already flushing 2023-04-12 15:54:49,765 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:49,765 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:49,765 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:49,765 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,765 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:49,765 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:49,766 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:49,766 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:49,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:49,766 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:49,766 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:49,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:49,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:49,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:49,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:49,768 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:49,768 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ff3694d[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:49,768 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:49,768 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:49,769 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:49,769 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=249 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/A/fc27fa1ef4de4c7abf36e7c61dd079e3 2023-04-12 15:54:49,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:49,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:49,769 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:49,769 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:49,769 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:49,770 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,770 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:49,771 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,771 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:49,771 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:49,771 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:49,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:49,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:49,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:49,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:49,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:49,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:49,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:49,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:49,773 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:49,773 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:49,773 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:49,773 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:49,773 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:49,773 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:49,776 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:49,776 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:49,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:49,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:49,776 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:49,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:49,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:49,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:49,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:49,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:49,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:49,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:49,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:49,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:49,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:49,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,778 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:49,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:49,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,779 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,781 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:49,782 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:49,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:49,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:49,782 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,782 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:49,782 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:49,782 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:49,782 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:49,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:49,782 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:49,782 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:49,782 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:49,782 DEBUG [Thread-1647] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:49,782 DEBUG [Thread-1647] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:49,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:49,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:49,784 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=249 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/B/0bb7d3bc29a04316978f7260fb78b780 2023-04-12 15:54:49,882 DEBUG [Thread-1647] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:49,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:49,884 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:49,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:49,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:49,886 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:49,886 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:49,886 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:49,886 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:49,886 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:49,886 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:49,887 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:49,887 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:49,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:49,888 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,888 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:49,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:49,888 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:49,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:49,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:49,889 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:49,889 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:49,889 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:49,889 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:49,889 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:49,889 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:49,889 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:49,889 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:49,889 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:49,890 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. started... 2023-04-12 15:54:49,890 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. as already flushing 2023-04-12 15:54:49,890 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:49,890 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:49,890 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:49,890 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,890 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:49,890 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:49,891 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:49,891 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:49,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:49,892 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:49,892 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:49,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:49,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:49,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:49,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:49,893 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:49,893 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a4f2d2f[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:49,893 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:49,893 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:49,895 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:49,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:49,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:49,895 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,895 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:49,895 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:49,895 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:49,895 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:49,896 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,896 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:49,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:49,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:49,896 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:49,897 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:49,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:49,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:49,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:49,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:49,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:49,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:49,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:49,899 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:49,899 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:49,899 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:49,899 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:49,899 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:49,900 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:49,900 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:49,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:49,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:49,900 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:49,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:49,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:49,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:49,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:49,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:49,901 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:49,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:49,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:49,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:49,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:49,902 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:49,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:49,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,904 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,904 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:49,904 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:49,904 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:49,904 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:49,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:49,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:49,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:49,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:49,904 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:49,904 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:49,904 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:49,904 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:49,905 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:49,905 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:49,904 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:49,905 DEBUG [Thread-1647] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:49,905 DEBUG [Thread-1647] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:49,905 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:50,005 DEBUG [Thread-1647] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:50,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:50,007 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:50,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:50,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:50,008 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:50,008 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:50,008 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:50,008 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:50,009 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:50,009 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:50,011 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,011 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:50,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:50,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:50,011 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,011 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:50,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:50,011 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:50,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:50,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:50,012 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:50,012 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:50,012 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:50,012 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:50,012 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:50,012 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:50,012 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:50,012 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:50,012 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:50,012 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. started... 2023-04-12 15:54:50,012 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. as already flushing 2023-04-12 15:54:50,012 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:50,013 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:50,013 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:50,013 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,013 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:50,013 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:50,014 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,014 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:50,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:50,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:50,015 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:50,015 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:50,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:50,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:50,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:50,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:50,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:50,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@fc402e4[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:50,016 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:50,016 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:50,017 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:50,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:50,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:50,017 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:50,017 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:50,017 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:50,017 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,017 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:50,018 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,018 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:50,018 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:50,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:50,018 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:50,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:50,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:50,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:50,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:50,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:50,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:50,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:50,021 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:50,021 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:50,021 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:50,021 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:50,021 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:50,021 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:50,025 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:50,025 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:50,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:50,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:50,025 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:50,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:50,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:50,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:50,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:50,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:50,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:50,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:50,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:50,026 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:50,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:50,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,027 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:50,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:50,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,029 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,029 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:50,029 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:50,029 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:50,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:50,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:50,029 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:50,029 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:50,029 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:50,030 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:50,029 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,030 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:50,030 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:50,030 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:50,030 DEBUG [Thread-1647] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:50,030 DEBUG [Thread-1647] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:50,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:50,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:50,050 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:50,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53040 deadline: 1681314950048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,053 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:50,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53012 deadline: 1681314950051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,053 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:50,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53014 deadline: 1681314950053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,054 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:50,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53032 deadline: 1681314950053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,056 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:50,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53078 deadline: 1681314950056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,130 DEBUG [Thread-1647] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:50,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:50,132 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:50,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:50,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:50,133 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:50,133 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:50,133 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:50,133 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:50,133 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:50,133 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:50,135 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:50,135 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:50,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:50,135 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,135 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:50,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:50,136 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:50,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:50,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:50,136 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:50,136 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:50,136 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:50,136 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:50,137 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:50,137 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:50,137 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:50,137 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:50,137 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:50,137 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. started... 2023-04-12 15:54:50,137 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. as already flushing 2023-04-12 15:54:50,137 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:50,137 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:50,137 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:50,137 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,137 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:50,137 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:50,142 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:50,142 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:50,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:50,142 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:50,142 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:50,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:50,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:50,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:50,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:50,143 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:50,143 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a2c34be[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:50,143 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:50,143 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:50,145 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:50,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:50,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:50,145 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,145 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:50,145 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:50,145 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:50,145 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:50,147 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,147 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:50,147 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:50,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:50,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:50,147 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:50,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:50,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:50,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:50,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:50,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:50,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:50,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:50,149 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:50,149 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:50,149 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:50,149 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:50,149 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:50,151 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:50,151 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:50,151 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:50,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:50,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:50,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:50,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:50,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:50,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:50,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:50,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:50,152 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:50,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:50,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:50,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:50,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,154 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:50,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:50,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,158 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:50,158 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:50,158 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,158 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:50,158 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:50,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:50,158 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:50,158 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:50,158 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:50,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:50,158 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,159 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:50,159 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:50,159 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:50,159 DEBUG [Thread-1647] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:50,159 DEBUG [Thread-1647] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:50,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:50,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:50,198 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=249 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/C/0e2cffc2d94e4c2b9682d16e62c759e1 2023-04-12 15:54:50,202 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/A/fc27fa1ef4de4c7abf36e7c61dd079e3 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/fc27fa1ef4de4c7abf36e7c61dd079e3 2023-04-12 15:54:50,207 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/fc27fa1ef4de4c7abf36e7c61dd079e3, entries=200, sequenceid=249, filesize=14.1 K 2023-04-12 15:54:50,207 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/B/0bb7d3bc29a04316978f7260fb78b780 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/0bb7d3bc29a04316978f7260fb78b780 2023-04-12 15:54:50,212 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/0bb7d3bc29a04316978f7260fb78b780, entries=150, sequenceid=249, filesize=11.8 K 2023-04-12 15:54:50,212 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/C/0e2cffc2d94e4c2b9682d16e62c759e1 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/0e2cffc2d94e4c2b9682d16e62c759e1 2023-04-12 15:54:50,217 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/0e2cffc2d94e4c2b9682d16e62c759e1, entries=150, sequenceid=249, filesize=11.8 K 2023-04-12 15:54:50,218 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for deda1ce51fef7ef8d26da255318c55e4 in 859ms, sequenceid=249, compaction requested=true 2023-04-12 15:54:50,218 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for deda1ce51fef7ef8d26da255318c55e4: 2023-04-12 15:54:50,218 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-12 15:54:50,218 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:54:50,218 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-12 15:54:50,218 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:54:50,218 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:54:50,220 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 53266 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-12 15:54:50,220 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1912): deda1ce51fef7ef8d26da255318c55e4/A is initiating minor compaction (all files) 2023-04-12 15:54:50,220 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of deda1ce51fef7ef8d26da255318c55e4/A in TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:50,220 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/719ca553c87d49ae9ef9e7f4a0507870, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/7295be13e3ff4465866d021d82f31e76, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/b654ff237113487d8e5a691b7c9c0d94, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/fc27fa1ef4de4c7abf36e7c61dd079e3] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp, totalSize=52.0 K 2023-04-12 15:54:50,220 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48486 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-12 15:54:50,221 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1912): deda1ce51fef7ef8d26da255318c55e4/B is initiating minor compaction (all files) 2023-04-12 15:54:50,221 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 719ca553c87d49ae9ef9e7f4a0507870, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1681314887279 2023-04-12 15:54:50,221 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2259): Starting compaction of deda1ce51fef7ef8d26da255318c55e4/B in TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:50,221 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/9a066fa7382641d1ac5d8d9609b9ca03, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/27cc961916fc42658d3d2f3d0be19021, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/cf1523cb7d924a47b293492713c6b0cc, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/0bb7d3bc29a04316978f7260fb78b780] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp, totalSize=47.3 K 2023-04-12 15:54:50,221 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 7295be13e3ff4465866d021d82f31e76, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=222, earliestPutTs=1681314887290 2023-04-12 15:54:50,221 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 9a066fa7382641d1ac5d8d9609b9ca03, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1681314887279 2023-04-12 15:54:50,222 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting b654ff237113487d8e5a691b7c9c0d94, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=238, earliestPutTs=1681314888221 2023-04-12 15:54:50,222 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 27cc961916fc42658d3d2f3d0be19021, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=222, earliestPutTs=1681314887290 2023-04-12 15:54:50,222 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting fc27fa1ef4de4c7abf36e7c61dd079e3, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=249, earliestPutTs=1681314889348 2023-04-12 15:54:50,222 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting cf1523cb7d924a47b293492713c6b0cc, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=238, earliestPutTs=1681314888221 2023-04-12 15:54:50,222 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 0bb7d3bc29a04316978f7260fb78b780, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=249, earliestPutTs=1681314889348 2023-04-12 15:54:50,230 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] throttle.PressureAwareThroughputController(145): deda1ce51fef7ef8d26da255318c55e4#B#compaction#244 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:54:50,230 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] throttle.PressureAwareThroughputController(145): deda1ce51fef7ef8d26da255318c55e4#A#compaction#243 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:54:50,242 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/B/bd59478b2ea34208b7f43f15c90f57fe as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/bd59478b2ea34208b7f43f15c90f57fe 2023-04-12 15:54:50,247 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in deda1ce51fef7ef8d26da255318c55e4/B of deda1ce51fef7ef8d26da255318c55e4 into bd59478b2ea34208b7f43f15c90f57fe(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:54:50,247 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for deda1ce51fef7ef8d26da255318c55e4: 2023-04-12 15:54:50,247 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4., storeName=deda1ce51fef7ef8d26da255318c55e4/B, priority=12, startTime=1681314890218; duration=0sec 2023-04-12 15:54:50,247 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:54:50,248 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-12 15:54:50,249 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48486 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-12 15:54:50,249 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1912): deda1ce51fef7ef8d26da255318c55e4/C is initiating minor compaction (all files) 2023-04-12 15:54:50,249 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2259): Starting compaction of deda1ce51fef7ef8d26da255318c55e4/C in TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:50,249 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/754391c76d23499b9475cc8a247ace5e, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/d43f8ae378264d50b474ef7c7fbf544e, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/88bd2a2da3b040df9ea5948d729d9742, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/0e2cffc2d94e4c2b9682d16e62c759e1] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp, totalSize=47.3 K 2023-04-12 15:54:50,249 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 754391c76d23499b9475cc8a247ace5e, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1681314887279 2023-04-12 15:54:50,250 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting d43f8ae378264d50b474ef7c7fbf544e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=222, earliestPutTs=1681314887290 2023-04-12 15:54:50,250 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 88bd2a2da3b040df9ea5948d729d9742, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=238, earliestPutTs=1681314888221 2023-04-12 15:54:50,250 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 0e2cffc2d94e4c2b9682d16e62c759e1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=249, earliestPutTs=1681314889348 2023-04-12 15:54:50,258 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] throttle.PressureAwareThroughputController(145): deda1ce51fef7ef8d26da255318c55e4#C#compaction#245 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:54:50,259 DEBUG [Thread-1647] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:50,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:50,261 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:50,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:50,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:50,266 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:50,266 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:50,266 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:50,267 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:50,267 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:50,267 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:50,268 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:50,268 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:50,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:50,268 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,269 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:50,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:50,269 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:50,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:50,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:50,270 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:50,270 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:50,270 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:50,270 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:50,270 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:50,270 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:50,270 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:50,270 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:50,270 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. started... 2023-04-12 15:54:50,271 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing deda1ce51fef7ef8d26da255318c55e4 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-04-12 15:54:50,271 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK deda1ce51fef7ef8d26da255318c55e4, store=A 2023-04-12 15:54:50,271 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:54:50,271 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK deda1ce51fef7ef8d26da255318c55e4, store=B 2023-04-12 15:54:50,271 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:54:50,272 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK deda1ce51fef7ef8d26da255318c55e4, store=C 2023-04-12 15:54:50,272 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:54:50,274 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/C/da462dd7b00c46b09995e94cfd2a9cf9 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/da462dd7b00c46b09995e94cfd2a9cf9 2023-04-12 15:54:50,278 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in deda1ce51fef7ef8d26da255318c55e4/C of deda1ce51fef7ef8d26da255318c55e4 into da462dd7b00c46b09995e94cfd2a9cf9(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:54:50,278 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for deda1ce51fef7ef8d26da255318c55e4: 2023-04-12 15:54:50,278 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4., storeName=deda1ce51fef7ef8d26da255318c55e4/C, priority=12, startTime=1681314890218; duration=0sec 2023-04-12 15:54:50,279 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:54:50,284 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=287 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/A/622e1d87c7084859b72cd6c4fada624a 2023-04-12 15:54:50,294 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=287 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/B/408dec5f6a6a4bc3b657548399504c21 2023-04-12 15:54:50,302 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=287 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/C/7c790d6efe1640ab8f931b59aed17680 2023-04-12 15:54:50,306 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/A/622e1d87c7084859b72cd6c4fada624a as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/622e1d87c7084859b72cd6c4fada624a 2023-04-12 15:54:50,309 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/622e1d87c7084859b72cd6c4fada624a, entries=150, sequenceid=287, filesize=11.9 K 2023-04-12 15:54:50,310 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/B/408dec5f6a6a4bc3b657548399504c21 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/408dec5f6a6a4bc3b657548399504c21 2023-04-12 15:54:50,313 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/408dec5f6a6a4bc3b657548399504c21, entries=150, sequenceid=287, filesize=11.9 K 2023-04-12 15:54:50,314 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/C/7c790d6efe1640ab8f931b59aed17680 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/7c790d6efe1640ab8f931b59aed17680 2023-04-12 15:54:50,319 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/7c790d6efe1640ab8f931b59aed17680, entries=150, sequenceid=287, filesize=11.9 K 2023-04-12 15:54:50,320 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=0 B/0 for deda1ce51fef7ef8d26da255318c55e4 in 49ms, sequenceid=287, compaction requested=false 2023-04-12 15:54:50,320 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for deda1ce51fef7ef8d26da255318c55e4: 2023-04-12 15:54:50,320 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:50,320 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:50,320 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:50,320 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,324 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:50,324 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:50,325 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:50,325 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:50,325 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:50,326 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:50,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:50,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:50,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:50,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:50,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:50,328 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:50,328 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ee18fb7[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:50,328 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:50,328 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:50,329 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:50,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:50,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:50,330 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:50,330 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:50,330 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:50,330 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,330 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:50,332 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,332 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:50,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:50,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:50,332 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:50,332 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:50,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:50,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:50,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:50,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:50,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:50,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:50,335 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:50,335 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:50,335 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:50,335 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:50,335 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:50,335 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:50,336 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:50,336 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:50,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:50,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:50,336 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:50,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:50,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:50,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:50,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:50,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:50,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:50,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:50,337 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:50,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:50,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:50,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:50,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:50,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,340 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,342 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,342 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:50,342 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:50,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:50,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:50,342 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:50,343 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:50,343 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:50,343 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:50,343 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:50,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:50,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:50,343 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,343 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:50,343 DEBUG [Thread-1647] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:50,343 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:50,343 DEBUG [Thread-1647] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:50,343 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:50,443 DEBUG [Thread-1647] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:50,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:50,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:50,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:50,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:50,447 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:50,447 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:50,447 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:50,447 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:50,447 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:50,447 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:50,448 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:50,448 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:50,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:50,449 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,449 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:50,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:50,449 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:50,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:50,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:50,450 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:50,450 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:50,450 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:50,450 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:50,450 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:50,450 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:50,450 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:50,451 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:50,451 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:50,451 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. started... 2023-04-12 15:54:50,451 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for deda1ce51fef7ef8d26da255318c55e4: 2023-04-12 15:54:50,451 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:50,451 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:50,451 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:50,451 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,451 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:50,451 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:50,453 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,453 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:50,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:50,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:50,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:50,454 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:50,454 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:50,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:50,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:50,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:50,455 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:50,455 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d64f111[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:50,455 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:50,455 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:50,456 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:50,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:50,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:50,456 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:50,456 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:50,456 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:50,457 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,457 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:50,458 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,458 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:50,458 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:50,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:50,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:50,458 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:50,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:50,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:50,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:50,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:50,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:50,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:50,460 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:50,460 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:50,460 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:50,460 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:50,460 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:50,460 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:50,462 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:50,462 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:50,462 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:50,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:50,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:50,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:50,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:50,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:50,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:50,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:50,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:50,463 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:50,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:50,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:50,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:50,464 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:50,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:50,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,467 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,467 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:50,467 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:50,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:50,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:50,467 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:50,467 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:50,467 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:50,467 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:50,467 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:50,467 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,467 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:50,467 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:50,467 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:50,467 DEBUG [Thread-1647] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:50,467 DEBUG [Thread-1647] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:50,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:50,468 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:50,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:50,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on deda1ce51fef7ef8d26da255318c55e4 2023-04-12 15:54:50,559 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing deda1ce51fef7ef8d26da255318c55e4 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-12 15:54:50,560 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK deda1ce51fef7ef8d26da255318c55e4, store=A 2023-04-12 15:54:50,560 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:50,560 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK deda1ce51fef7ef8d26da255318c55e4, store=B 2023-04-12 15:54:50,560 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:50,560 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK deda1ce51fef7ef8d26da255318c55e4, store=C 2023-04-12 15:54:50,560 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:50,568 DEBUG [Thread-1647] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:50,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:50,572 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:50,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:50,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:50,575 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=300 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/A/83a439098e854e14bbedf4374ebb533e 2023-04-12 15:54:50,575 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:50,578 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:50,578 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:50,578 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:50,578 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:50,578 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:50,579 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:50,580 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:50,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:50,580 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,580 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:50,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:50,580 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:50,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:50,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:50,581 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:50,581 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:50,581 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:50,581 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:50,581 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:50,581 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:50,581 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:50,582 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:50,582 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:50,582 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. started... 2023-04-12 15:54:50,582 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. as already flushing 2023-04-12 15:54:50,582 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:50,582 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:50,582 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:50,582 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,582 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:50,582 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:50,583 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,583 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:50,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:50,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:50,584 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:50,584 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:50,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:50,584 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:50,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53014 deadline: 1681314950581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:50,584 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:50,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53032 deadline: 1681314950581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,585 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681314890568, max timestamp=1681314890583Num uniques 859; 2023-04-12 15:54:50,585 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-12 15:54:50,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:50,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:50,586 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:50,586 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681314890568, max timestamp=1681314890583Num uniques 859; 2023-04-12 15:54:50,586 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:50,586 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-12 15:54:50,586 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6298197b[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:50,586 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:50,587 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681314890568, max timestamp=1681314890583Num uniques 859; 2023-04-12 15:54:50,587 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-12 15:54:50,587 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:50,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:50,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:50,587 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:50,587 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:50,587 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:50,588 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,588 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:50,592 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:50,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:50,592 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:50,593 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:50,593 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:50,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:50,593 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=300 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/B/f5a0f2a16ee04a6f9e61f8dc971ce7d5 2023-04-12 15:54:50,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:50,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:50,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:50,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:50,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:50,595 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:50,595 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:50,595 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:50,595 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:50,595 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:50,595 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:50,597 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:50,597 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:50,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:50,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:50,597 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:50,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:50,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:50,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:50,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:50,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:50,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:50,598 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:50,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:50,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:50,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:50,599 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:50,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:50,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,602 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,602 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:50,602 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:50,602 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:50,602 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:50,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:50,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:50,602 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:50,602 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:50,603 DEBUG [Thread-1647] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:50,603 DEBUG [Thread-1647] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:50,602 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:50,603 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,603 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:50,603 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:50,603 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:50,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:50,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:50,603 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=300 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/C/acc425b0bdf54d729d7d7a448c996ac9 2023-04-12 15:54:50,605 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:50,607 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/A/83a439098e854e14bbedf4374ebb533e as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/83a439098e854e14bbedf4374ebb533e 2023-04-12 15:54:50,611 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/83a439098e854e14bbedf4374ebb533e, entries=200, sequenceid=300, filesize=14.3 K 2023-04-12 15:54:50,611 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/B/f5a0f2a16ee04a6f9e61f8dc971ce7d5 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/f5a0f2a16ee04a6f9e61f8dc971ce7d5 2023-04-12 15:54:50,615 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:50,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53012 deadline: 1681314950608, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,617 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/f5a0f2a16ee04a6f9e61f8dc971ce7d5, entries=150, sequenceid=300, filesize=11.9 K 2023-04-12 15:54:50,617 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:50,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53040 deadline: 1681314950609, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,618 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/C/acc425b0bdf54d729d7d7a448c996ac9 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/acc425b0bdf54d729d7d7a448c996ac9 2023-04-12 15:54:50,618 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:50,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53078 deadline: 1681314950615, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,623 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/acc425b0bdf54d729d7d7a448c996ac9, entries=150, sequenceid=300, filesize=11.9 K 2023-04-12 15:54:50,624 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for deda1ce51fef7ef8d26da255318c55e4 in 65ms, sequenceid=300, compaction requested=true 2023-04-12 15:54:50,624 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for deda1ce51fef7ef8d26da255318c55e4: 2023-04-12 15:54:50,624 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:54:50,624 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:54:50,624 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 4 compacting, 2 eligible, 16 blocking 2023-04-12 15:54:50,624 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-12 15:54:50,625 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-12 15:54:50,625 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-12 15:54:50,625 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. because compaction request was cancelled 2023-04-12 15:54:50,625 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 15:54:50,626 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36886 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 15:54:50,626 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1912): deda1ce51fef7ef8d26da255318c55e4/B is initiating minor compaction (all files) 2023-04-12 15:54:50,626 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2259): Starting compaction of deda1ce51fef7ef8d26da255318c55e4/B in TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:50,626 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/bd59478b2ea34208b7f43f15c90f57fe, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/408dec5f6a6a4bc3b657548399504c21, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/f5a0f2a16ee04a6f9e61f8dc971ce7d5] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp, totalSize=36.0 K 2023-04-12 15:54:50,626 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting bd59478b2ea34208b7f43f15c90f57fe, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=249, earliestPutTs=1681314889348 2023-04-12 15:54:50,627 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 408dec5f6a6a4bc3b657548399504c21, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=287, earliestPutTs=1681314889439 2023-04-12 15:54:50,627 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting f5a0f2a16ee04a6f9e61f8dc971ce7d5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=300, earliestPutTs=1681314890558 2023-04-12 15:54:50,633 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] throttle.PressureAwareThroughputController(145): deda1ce51fef7ef8d26da255318c55e4#B#compaction#252 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:54:50,643 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/A/f02f623dc7ef4913a53b0f3c8bcded12 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/f02f623dc7ef4913a53b0f3c8bcded12 2023-04-12 15:54:50,646 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in deda1ce51fef7ef8d26da255318c55e4/A of deda1ce51fef7ef8d26da255318c55e4 into f02f623dc7ef4913a53b0f3c8bcded12(size=12.2 K), total size for store is 38.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:54:50,646 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for deda1ce51fef7ef8d26da255318c55e4: 2023-04-12 15:54:50,646 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4., storeName=deda1ce51fef7ef8d26da255318c55e4/A, priority=12, startTime=1681314890218; duration=0sec 2023-04-12 15:54:50,646 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:54:50,646 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 15:54:50,647 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36886 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 15:54:50,647 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1912): deda1ce51fef7ef8d26da255318c55e4/C is initiating minor compaction (all files) 2023-04-12 15:54:50,647 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of deda1ce51fef7ef8d26da255318c55e4/C in TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:50,647 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/da462dd7b00c46b09995e94cfd2a9cf9, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/7c790d6efe1640ab8f931b59aed17680, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/acc425b0bdf54d729d7d7a448c996ac9] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp, totalSize=36.0 K 2023-04-12 15:54:50,648 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting da462dd7b00c46b09995e94cfd2a9cf9, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=249, earliestPutTs=1681314889348 2023-04-12 15:54:50,648 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 7c790d6efe1640ab8f931b59aed17680, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=287, earliestPutTs=1681314889439 2023-04-12 15:54:50,648 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting acc425b0bdf54d729d7d7a448c996ac9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=300, earliestPutTs=1681314890558 2023-04-12 15:54:50,653 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] throttle.PressureAwareThroughputController(145): deda1ce51fef7ef8d26da255318c55e4#C#compaction#253 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:54:50,664 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/C/c13908f0ee9943d382d20e68abf2a984 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/c13908f0ee9943d382d20e68abf2a984 2023-04-12 15:54:50,668 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in deda1ce51fef7ef8d26da255318c55e4/C of deda1ce51fef7ef8d26da255318c55e4 into c13908f0ee9943d382d20e68abf2a984(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:54:50,668 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for deda1ce51fef7ef8d26da255318c55e4: 2023-04-12 15:54:50,668 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4., storeName=deda1ce51fef7ef8d26da255318c55e4/C, priority=13, startTime=1681314890624; duration=0sec 2023-04-12 15:54:50,669 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:54:50,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on deda1ce51fef7ef8d26da255318c55e4 2023-04-12 15:54:50,687 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing deda1ce51fef7ef8d26da255318c55e4 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-12 15:54:50,687 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK deda1ce51fef7ef8d26da255318c55e4, store=A 2023-04-12 15:54:50,687 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:54:50,687 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK deda1ce51fef7ef8d26da255318c55e4, store=B 2023-04-12 15:54:50,687 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:54:50,687 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK deda1ce51fef7ef8d26da255318c55e4, store=C 2023-04-12 15:54:50,687 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:54:50,700 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:50,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53014 deadline: 1681314950697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,700 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:50,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53032 deadline: 1681314950698, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,703 DEBUG [Thread-1647] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:50,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:50,705 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:50,706 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=338 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/A/3dd684c350e34cb98a64f66f18f950b0 2023-04-12 15:54:50,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:50,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:50,707 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:50,707 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:50,707 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:50,707 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:50,707 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:50,707 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:50,709 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,709 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:50,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:50,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:50,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:50,709 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,709 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:50,710 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:50,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:50,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:50,710 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:50,710 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:50,710 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:50,710 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:50,710 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:50,710 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:50,711 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:50,711 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:50,711 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:50,711 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. started... 2023-04-12 15:54:50,711 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. as already flushing 2023-04-12 15:54:50,711 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:50,711 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:50,711 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:50,711 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,711 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:50,711 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:50,714 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:50,714 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:50,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:50,715 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:50,715 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:50,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:50,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:50,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:50,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:50,716 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:50,716 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@196a4852[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:50,716 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:50,716 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:50,717 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:50,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:50,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:50,718 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:50,718 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:50,718 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:50,718 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,718 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:50,718 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:50,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53012 deadline: 1681314950716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,719 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:50,719 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:50,719 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,719 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:50,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:50,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:50,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:50,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:50,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:50,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:50,721 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:50,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:50,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53040 deadline: 1681314950718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,722 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:50,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53078 deadline: 1681314950721, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:50,723 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:50,723 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:50,723 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:50,723 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:50,723 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:50,723 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:50,724 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:50,724 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:50,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:50,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:50,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:50,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:50,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:50,724 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:50,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:50,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:50,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:50,725 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:50,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:50,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:50,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:50,726 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:50,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:50,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,729 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:50,729 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,729 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:50,729 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:50,729 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:50,729 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:50,729 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:50,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:50,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:50,729 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:50,729 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:50,729 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,729 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:50,729 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:50,729 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:50,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:50,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:50,730 DEBUG [Thread-1647] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:50,730 DEBUG [Thread-1647] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:50,803 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:50,803 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:50,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53014 deadline: 1681314950801, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53032 deadline: 1681314950801, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,830 DEBUG [Thread-1647] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:50,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:50,832 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:50,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:50,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:50,833 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:50,833 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:50,833 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:50,833 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:50,836 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:50,836 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:50,837 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,837 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:50,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:50,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:50,837 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,837 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:50,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:50,838 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:50,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:50,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:50,838 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:50,838 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:50,838 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:50,838 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:50,839 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:50,839 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:50,839 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:50,839 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:50,839 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:50,839 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. started... 2023-04-12 15:54:50,839 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. as already flushing 2023-04-12 15:54:50,839 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:50,839 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:50,839 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:50,839 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,839 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:50,839 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:50,840 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,840 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:50,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:50,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:50,841 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:50,841 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:50,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:50,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:50,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:50,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:50,842 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:50,842 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b94f0ff[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:50,842 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:50,842 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:50,843 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:50,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:50,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:50,843 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:50,843 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:50,843 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:50,843 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,843 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:50,848 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,848 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:50,848 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:50,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:50,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:50,848 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:50,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:50,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:50,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:50,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:50,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:50,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:50,851 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:50,851 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:50,851 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:50,851 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:50,851 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:50,851 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:50,853 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:50,853 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:50,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:50,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:50,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:50,853 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:50,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:50,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:50,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:50,853 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:50,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:50,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:50,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:50,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:50,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:50,854 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:50,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:50,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,856 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,856 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:50,856 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:50,856 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:50,856 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:50,856 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:50,856 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:50,856 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,857 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:50,857 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:50,857 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:50,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:50,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:50,857 DEBUG [Thread-1647] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:50,856 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:50,857 DEBUG [Thread-1647] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:50,856 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:50,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:50,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:50,921 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:50,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53012 deadline: 1681314950919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,924 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:50,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53040 deadline: 1681314950923, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,926 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:50,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53078 deadline: 1681314950924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,957 DEBUG [Thread-1647] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:50,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:50,959 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:50,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:50,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:50,960 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:50,960 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:50,960 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:50,960 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:50,960 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:50,960 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:50,961 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,961 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:50,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:50,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:50,962 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,962 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:50,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:50,962 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:50,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:50,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:50,963 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:50,979 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:50,979 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:50,979 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:50,979 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:50,979 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:50,979 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:50,979 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:50,980 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:50,980 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. started... 2023-04-12 15:54:50,980 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. as already flushing 2023-04-12 15:54:50,980 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:50,980 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:50,980 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:50,980 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,980 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:50,980 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:50,982 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,983 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:50,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:50,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:50,983 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:50,983 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:50,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:50,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:50,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:50,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:50,985 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:50,985 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11927ae5[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:50,985 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:50,985 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:50,986 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:50,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:50,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:50,986 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:50,986 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:50,986 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,986 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:50,986 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:50,988 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,988 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:50,988 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:50,988 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:50,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:50,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:50,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:50,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:50,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:50,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:50,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:50,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:50,990 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:50,990 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:50,990 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:50,990 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:50,990 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:50,991 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:50,995 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:50,995 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:50,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:50,995 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:50,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:50,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:50,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:50,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:50,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:50,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:50,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:50,996 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:50,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:50,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:50,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:50,997 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:50,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:50,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:50,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,001 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,001 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:51,001 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:51,001 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:51,001 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:51,001 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:51,001 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,001 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:51,001 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:51,001 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:51,001 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:51,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:51,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:51,001 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:51,001 DEBUG [Thread-1647] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:51,001 DEBUG [Thread-1647] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:51,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:51,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:51,005 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:51,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53014 deadline: 1681314951004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,005 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:51,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53032 deadline: 1681314951005, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,044 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/B/32c926e62d6f4400b6a6d14109b5b287 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/32c926e62d6f4400b6a6d14109b5b287 2023-04-12 15:54:51,048 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in deda1ce51fef7ef8d26da255318c55e4/B of deda1ce51fef7ef8d26da255318c55e4 into 32c926e62d6f4400b6a6d14109b5b287(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:54:51,048 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for deda1ce51fef7ef8d26da255318c55e4: 2023-04-12 15:54:51,048 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4., storeName=deda1ce51fef7ef8d26da255318c55e4/B, priority=13, startTime=1681314890624; duration=0sec 2023-04-12 15:54:51,048 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:54:51,102 DEBUG [Thread-1647] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:51,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:51,104 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:51,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:51,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:51,105 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:51,105 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:51,105 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:51,105 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:51,105 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:51,105 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:51,107 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:51,107 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:51,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:51,107 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:51,107 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:51,107 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:51,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:51,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:51,108 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:51,108 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:51,108 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:51,108 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:51,108 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:51,108 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:51,108 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:51,109 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:51,109 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:51,109 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. started... 2023-04-12 15:54:51,109 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. as already flushing 2023-04-12 15:54:51,109 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:51,109 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:51,109 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:51,109 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,109 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:51,109 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:51,110 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,110 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:51,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:51,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:51,111 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:51,111 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:51,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:51,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:51,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:51,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:51,112 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:51,112 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33e3ea6d[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:51,112 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:51,112 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:51,114 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:51,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:51,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:51,114 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:51,114 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,114 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:51,114 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:51,114 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:51,116 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:51,116 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,116 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:51,116 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:51,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:51,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:51,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:51,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:51,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:51,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:51,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:51,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:51,118 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:51,118 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:51,118 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:51,118 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:51,118 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:51,118 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:51,120 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:51,120 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:51,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:51,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:51,120 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:51,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:51,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:51,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:51,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:51,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:51,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:51,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:51,120 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:51,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:51,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:51,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,122 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,122 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=338 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/B/208bc335880a4650812d288c97195690 2023-04-12 15:54:51,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:51,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:51,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,127 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,127 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:51,127 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:51,127 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,127 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:51,127 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:51,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:51,127 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:51,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:51,127 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:51,127 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:51,127 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:51,127 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:51,127 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:51,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:51,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:51,128 DEBUG [Thread-1647] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:51,128 DEBUG [Thread-1647] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:51,223 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:51,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53012 deadline: 1681314951222, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,228 DEBUG [Thread-1647] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:51,228 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:51,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53040 deadline: 1681314951227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:51,230 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:51,231 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:51,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53078 deadline: 1681314951229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:51,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:51,231 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:51,231 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:51,231 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:51,231 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:51,232 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:51,232 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:51,234 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,234 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:51,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:51,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:51,234 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,234 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:51,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:51,235 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:51,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:51,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:51,235 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:51,235 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:51,235 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:51,235 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:51,235 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:51,235 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:51,235 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:51,235 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:51,235 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:51,235 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. started... 2023-04-12 15:54:51,235 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. as already flushing 2023-04-12 15:54:51,235 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:51,236 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:51,236 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:51,236 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,236 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:51,236 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:51,237 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,237 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:51,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:51,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:51,238 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:51,238 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:51,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:51,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:51,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:51,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:51,239 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:51,239 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78001c9b[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:51,239 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:51,239 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:51,240 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:51,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:51,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:51,240 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:51,240 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:51,240 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:51,240 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,240 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:51,241 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,241 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:51,241 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:51,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:51,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:51,241 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:51,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:51,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:51,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:51,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:51,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:51,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:51,243 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:51,243 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:51,243 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:51,243 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:51,243 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:51,243 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:51,246 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:51,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:51,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:51,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:51,246 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:51,246 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:51,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:51,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:51,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:51,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:51,246 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:51,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:51,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:51,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:51,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:51,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,248 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:51,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:51,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,250 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:51,250 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,250 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:51,250 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:51,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:51,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:51,250 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:51,250 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:51,250 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:51,250 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:51,250 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,250 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:51,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:51,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:51,250 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:51,250 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:51,250 DEBUG [Thread-1647] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:51,250 DEBUG [Thread-1647] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:51,309 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:51,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53014 deadline: 1681314951306, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,309 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:51,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53032 deadline: 1681314951306, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,350 DEBUG [Thread-1647] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:51,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:51,352 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:51,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:51,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:51,353 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:51,353 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:51,354 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:51,354 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:51,354 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:51,354 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:51,355 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,355 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:51,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:51,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:51,355 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,355 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:51,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:51,356 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:51,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:51,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:51,356 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:51,356 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:51,356 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:51,357 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:51,357 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:51,357 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:51,357 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:51,357 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:51,357 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:51,357 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. started... 2023-04-12 15:54:51,357 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. as already flushing 2023-04-12 15:54:51,357 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:51,357 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:51,357 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:51,357 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,357 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:51,357 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:51,358 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:51,358 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:51,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:51,359 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:51,359 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:51,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:51,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:51,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:51,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:51,360 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:51,360 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@244f186c[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:51,360 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:51,360 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:51,362 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:51,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:51,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:51,362 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:51,362 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:51,362 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:51,363 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,363 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:51,364 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:51,364 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:51,364 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,364 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:51,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:51,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:51,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:51,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:51,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:51,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:51,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:51,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:51,366 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:51,366 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:51,366 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:51,366 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:51,367 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:51,367 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:51,368 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:51,368 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:51,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:51,368 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:51,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:51,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:51,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:51,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:51,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:51,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:51,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:51,369 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:51,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:51,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:51,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:51,370 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:51,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:51,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,379 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,379 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:51,380 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:51,380 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:51,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:51,380 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:51,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:51,380 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:51,380 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:51,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:51,380 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:51,380 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:51,380 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:51,380 DEBUG [Thread-1647] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:51,380 DEBUG [Thread-1647] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:51,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:51,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:51,480 DEBUG [Thread-1647] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:51,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:51,482 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:51,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:51,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:51,484 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:51,484 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:51,484 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:51,484 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:51,484 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:51,484 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:51,486 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:51,486 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:51,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:51,486 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,486 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:51,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:51,487 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:51,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:51,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:51,487 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:51,487 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:51,487 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:51,487 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:51,487 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:51,487 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:51,487 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:51,487 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:51,488 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:51,488 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. started... 2023-04-12 15:54:51,488 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. as already flushing 2023-04-12 15:54:51,488 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:51,488 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:51,488 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:51,488 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,488 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:51,488 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:51,489 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,490 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:51,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:51,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:51,490 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:51,490 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:51,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:51,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:51,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:51,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:51,491 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:51,491 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26881b52[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:51,491 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:51,491 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:51,492 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:51,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:51,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:51,493 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:51,493 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:51,493 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:51,493 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,493 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:51,499 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,499 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:51,499 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:51,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:51,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:51,499 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:51,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:51,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:51,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:51,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:51,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:51,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:51,501 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:51,501 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:51,501 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:51,501 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:51,501 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:51,502 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:51,503 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:51,503 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:51,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:51,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:51,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:51,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:51,503 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:51,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:51,503 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:51,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:51,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:51,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:51,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:51,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:51,505 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:51,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:51,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:51,507 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,507 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:51,507 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:51,507 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,507 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:51,507 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:51,507 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:51,507 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:51,507 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:51,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:51,507 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:51,507 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:51,507 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:51,507 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:51,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:51,508 DEBUG [Thread-1647] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:51,508 DEBUG [Thread-1647] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:51,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:51,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:51,532 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=338 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/C/1ab3da9470f4491bae562852f0621a4d 2023-04-12 15:54:51,536 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/A/3dd684c350e34cb98a64f66f18f950b0 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/3dd684c350e34cb98a64f66f18f950b0 2023-04-12 15:54:51,539 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/3dd684c350e34cb98a64f66f18f950b0, entries=150, sequenceid=338, filesize=11.9 K 2023-04-12 15:54:51,540 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/B/208bc335880a4650812d288c97195690 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/208bc335880a4650812d288c97195690 2023-04-12 15:54:51,544 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/208bc335880a4650812d288c97195690, entries=150, sequenceid=338, filesize=11.9 K 2023-04-12 15:54:51,545 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/C/1ab3da9470f4491bae562852f0621a4d as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/1ab3da9470f4491bae562852f0621a4d 2023-04-12 15:54:51,548 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/1ab3da9470f4491bae562852f0621a4d, entries=150, sequenceid=338, filesize=11.9 K 2023-04-12 15:54:51,549 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for deda1ce51fef7ef8d26da255318c55e4 in 862ms, sequenceid=338, compaction requested=true 2023-04-12 15:54:51,549 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for deda1ce51fef7ef8d26da255318c55e4: 2023-04-12 15:54:51,549 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:54:51,549 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-12 15:54:51,549 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:54:51,549 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-12 15:54:51,549 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:54:51,550 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-12 15:54:51,550 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-12 15:54:51,550 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. because compaction request was cancelled 2023-04-12 15:54:51,550 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-12 15:54:51,551 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 51512 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-12 15:54:51,551 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1912): deda1ce51fef7ef8d26da255318c55e4/A is initiating minor compaction (all files) 2023-04-12 15:54:51,551 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of deda1ce51fef7ef8d26da255318c55e4/A in TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:51,551 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/f02f623dc7ef4913a53b0f3c8bcded12, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/622e1d87c7084859b72cd6c4fada624a, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/83a439098e854e14bbedf4374ebb533e, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/3dd684c350e34cb98a64f66f18f950b0] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp, totalSize=50.3 K 2023-04-12 15:54:51,551 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-12 15:54:51,551 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-12 15:54:51,551 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. because compaction request was cancelled 2023-04-12 15:54:51,551 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting f02f623dc7ef4913a53b0f3c8bcded12, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=249, earliestPutTs=1681314889348 2023-04-12 15:54:51,552 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 622e1d87c7084859b72cd6c4fada624a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=287, earliestPutTs=1681314889439 2023-04-12 15:54:51,552 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 83a439098e854e14bbedf4374ebb533e, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=300, earliestPutTs=1681314890556 2023-04-12 15:54:51,553 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 3dd684c350e34cb98a64f66f18f950b0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=338, earliestPutTs=1681314890604 2023-04-12 15:54:51,559 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] throttle.PressureAwareThroughputController(145): deda1ce51fef7ef8d26da255318c55e4#A#compaction#257 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:54:51,568 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/A/901263c6a73b48309269f92b6c0602f5 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/901263c6a73b48309269f92b6c0602f5 2023-04-12 15:54:51,572 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in deda1ce51fef7ef8d26da255318c55e4/A of deda1ce51fef7ef8d26da255318c55e4 into 901263c6a73b48309269f92b6c0602f5(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:54:51,572 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for deda1ce51fef7ef8d26da255318c55e4: 2023-04-12 15:54:51,572 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4., storeName=deda1ce51fef7ef8d26da255318c55e4/A, priority=12, startTime=1681314891549; duration=0sec 2023-04-12 15:54:51,573 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:54:51,608 DEBUG [Thread-1647] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:51,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:51,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:51,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:51,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:51,611 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:51,611 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:51,611 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:51,611 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:51,611 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:51,611 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:51,613 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:51,613 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:51,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:51,613 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,613 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:51,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:51,614 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:51,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:51,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:51,614 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:51,614 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:51,614 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:51,614 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:51,614 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:51,614 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:51,614 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:51,614 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:51,615 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:51,615 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. started... 2023-04-12 15:54:51,615 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing deda1ce51fef7ef8d26da255318c55e4 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-12 15:54:51,615 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK deda1ce51fef7ef8d26da255318c55e4, store=A 2023-04-12 15:54:51,615 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:51,615 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK deda1ce51fef7ef8d26da255318c55e4, store=B 2023-04-12 15:54:51,615 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:51,615 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK deda1ce51fef7ef8d26da255318c55e4, store=C 2023-04-12 15:54:51,615 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:51,622 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=350 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/A/f185d5def0e840c89b7cbab5ec98c899 2023-04-12 15:54:51,631 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=350 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/B/ca2496e21c8a4e6692f2780a74db43f0 2023-04-12 15:54:51,641 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=350 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/C/42507508ca7546ccab2ca456c438dc5f 2023-04-12 15:54:51,646 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/A/f185d5def0e840c89b7cbab5ec98c899 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/f185d5def0e840c89b7cbab5ec98c899 2023-04-12 15:54:51,650 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/f185d5def0e840c89b7cbab5ec98c899, entries=150, sequenceid=350, filesize=11.9 K 2023-04-12 15:54:51,651 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/B/ca2496e21c8a4e6692f2780a74db43f0 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/ca2496e21c8a4e6692f2780a74db43f0 2023-04-12 15:54:51,655 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/ca2496e21c8a4e6692f2780a74db43f0, entries=150, sequenceid=350, filesize=11.9 K 2023-04-12 15:54:51,655 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/C/42507508ca7546ccab2ca456c438dc5f as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/42507508ca7546ccab2ca456c438dc5f 2023-04-12 15:54:51,663 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/42507508ca7546ccab2ca456c438dc5f, entries=150, sequenceid=350, filesize=11.9 K 2023-04-12 15:54:51,664 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=0 B/0 for deda1ce51fef7ef8d26da255318c55e4 in 49ms, sequenceid=350, compaction requested=true 2023-04-12 15:54:51,664 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for deda1ce51fef7ef8d26da255318c55e4: 2023-04-12 15:54:51,664 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:51,664 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:51,664 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:51,664 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,664 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:51,664 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:51,667 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,667 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:51,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:51,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:51,668 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:51,668 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:51,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:51,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:51,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:51,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:51,669 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:51,669 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a142d6c[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:51,669 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:51,670 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:51,671 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:51,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:51,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:51,671 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:51,671 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:51,671 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,671 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:51,671 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:51,672 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:51,672 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,672 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:51,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:51,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:51,673 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:51,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:51,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:51,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:51,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:51,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:51,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:51,675 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:51,675 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:51,675 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:51,675 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:51,675 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:51,675 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:51,677 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:51,677 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:51,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:51,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:51,677 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:51,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:51,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:51,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:51,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:51,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:51,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:51,678 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:51,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:51,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:51,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:51,679 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:51,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:51,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,683 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:51,683 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:51,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:51,683 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:51,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:51,683 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:51,683 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:51,683 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,683 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:51,683 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:51,683 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,684 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:51,684 DEBUG [Thread-1647] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:51,684 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:51,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:51,684 DEBUG [Thread-1647] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:51,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:51,684 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:51,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on deda1ce51fef7ef8d26da255318c55e4 2023-04-12 15:54:51,736 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing deda1ce51fef7ef8d26da255318c55e4 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-12 15:54:51,738 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK deda1ce51fef7ef8d26da255318c55e4, store=A 2023-04-12 15:54:51,738 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:51,738 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK deda1ce51fef7ef8d26da255318c55e4, store=B 2023-04-12 15:54:51,738 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:51,738 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK deda1ce51fef7ef8d26da255318c55e4, store=C 2023-04-12 15:54:51,738 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:51,753 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=362 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/A/1f0c597b4e18471da07895fa82095be9 2023-04-12 15:54:51,784 DEBUG [Thread-1647] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:51,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:51,786 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:51,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:51,787 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:51,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:51,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53012 deadline: 1681314951784, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,787 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:51,787 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:51,787 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:51,787 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:51,787 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681314891743, max timestamp=1681314891786Num uniques 859; 2023-04-12 15:54:51,787 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-12 15:54:51,788 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681314891743, max timestamp=1681314891786Num uniques 859; 2023-04-12 15:54:51,788 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-12 15:54:51,788 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681314891743, max timestamp=1681314891786Num uniques 859; 2023-04-12 15:54:51,788 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:51,788 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:51,788 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-12 15:54:51,789 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,789 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:51,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:51,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:51,789 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,789 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:51,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:51,790 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:51,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:51,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:51,790 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:51,791 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:51,791 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:51,791 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:51,791 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:51,791 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:51,791 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:51,791 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:51,791 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:51,791 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. started... 2023-04-12 15:54:51,791 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. as already flushing 2023-04-12 15:54:51,791 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:51,791 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:51,791 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:51,791 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,791 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:51,791 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:51,793 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,793 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:51,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:51,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:51,793 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:51,793 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:51,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:51,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:51,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:51,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:51,794 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:51,795 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72080c71[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:51,795 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:51,795 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:51,796 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:51,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:51,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:51,796 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:51,796 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:51,796 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:51,796 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,796 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:51,798 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,798 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:51,798 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:51,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:51,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:51,798 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:51,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:51,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:51,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:51,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:51,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:51,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:51,801 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:51,801 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:51,801 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:51,801 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:51,801 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:51,801 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:51,802 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:51,802 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:51,802 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:51,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:51,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:51,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:51,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:51,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:51,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:51,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:51,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:51,803 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:51,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:51,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:51,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:51,804 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:51,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:51,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,806 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,806 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:51,806 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:51,806 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:51,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:51,806 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:51,806 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:51,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:51,806 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:51,807 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:51,806 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:51,807 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,807 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:51,807 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:51,807 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:51,807 DEBUG [Thread-1647] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:51,807 DEBUG [Thread-1647] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:51,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:51,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:51,814 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:51,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53032 deadline: 1681314951812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,817 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:51,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53078 deadline: 1681314951813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,817 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:51,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53014 deadline: 1681314951814, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,817 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:51,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53040 deadline: 1681314951814, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,890 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:51,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53012 deadline: 1681314951888, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,907 DEBUG [Thread-1647] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:51,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:51,909 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:51,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:51,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:51,910 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:51,910 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:51,910 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:51,910 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:51,911 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:51,911 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:51,919 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:51,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53078 deadline: 1681314951918, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,920 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:51,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53040 deadline: 1681314951918, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,920 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:51,920 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:51,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:51,920 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,920 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:51,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:51,921 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:51,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:51,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:51,921 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:51,921 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:51,921 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:51,921 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:51,921 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:51,921 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:51,922 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:51,922 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:51,922 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:51,922 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. started... 2023-04-12 15:54:51,922 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. as already flushing 2023-04-12 15:54:51,922 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:51,922 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:51,922 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:51,922 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,922 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:51,922 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:51,923 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,923 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:51,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:51,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:51,924 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:51,924 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:51,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:51,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:51,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:51,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:51,925 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:51,925 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e58e624[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:51,925 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:51,925 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:51,926 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:51,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:51,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:51,926 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:51,926 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:51,926 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,927 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:51,926 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:51,928 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,928 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:51,928 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:51,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:51,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:51,928 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:51,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:51,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:51,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:51,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:51,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:51,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:51,930 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:51,930 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:51,930 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:51,930 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:51,930 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:51,930 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:51,933 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:51,933 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:51,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:51,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:51,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:51,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:51,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:51,933 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:51,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:51,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:51,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:51,934 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:51,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:51,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:51,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:51,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,935 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:51,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:51,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,937 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,937 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:51,937 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:51,937 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:51,937 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:51,937 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:51,937 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:51,937 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:51,937 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:51,938 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:51,937 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:51,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:51,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:51,937 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:51,938 DEBUG [Thread-1647] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:51,938 DEBUG [Thread-1647] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:51,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:51,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:52,038 DEBUG [Thread-1647] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:52,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:52,040 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:52,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:52,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:52,041 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:52,041 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:52,041 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:52,041 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:52,042 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:52,042 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:52,043 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:52,043 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:52,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:52,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:52,043 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:52,043 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:52,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:52,043 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:52,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:52,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:52,044 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:52,044 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:52,044 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:52,044 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:52,044 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:52,044 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:52,044 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:52,044 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:52,044 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:52,044 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. started... 2023-04-12 15:54:52,044 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. as already flushing 2023-04-12 15:54:52,045 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:52,045 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:52,045 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:52,045 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:52,045 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:52,045 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:52,046 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:52,046 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:52,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:52,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:52,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:52,046 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:52,046 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:52,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:52,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:52,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:52,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:52,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:52,048 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:52,048 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@486c3a36[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:52,048 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:52,048 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:52,050 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:52,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:52,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:52,050 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:52,050 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:52,050 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:52,050 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:52,050 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:52,051 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:52,051 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:52,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:52,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:52,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:52,052 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:52,052 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:52,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:52,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:52,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:52,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:52,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:52,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:52,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:52,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:52,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:52,054 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:52,054 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:52,054 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:52,054 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:52,054 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:52,055 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:52,055 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:52,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:52,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:52,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:52,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:52,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:52,055 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:52,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:52,056 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:52,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:52,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:52,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:52,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:52,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:52,057 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:52,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:52,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:52,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:52,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:52,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:52,060 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:52,060 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:52,060 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:52,060 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:52,060 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:52,060 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:52,060 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:52,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:52,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:52,060 DEBUG [Thread-1647] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:52,060 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:52,060 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:52,060 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:52,060 DEBUG [Thread-1647] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:52,060 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:52,060 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:52,060 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:52,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:52,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:52,093 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:52,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53012 deadline: 1681314952091, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:52,123 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:52,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53078 deadline: 1681314952121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:52,123 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:52,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53040 deadline: 1681314952122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:52,160 DEBUG [Thread-1647] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:52,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:52,162 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:52,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:52,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:52,163 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:52,164 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:52,164 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:52,164 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:52,164 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:52,164 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:52,166 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:52,166 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:52,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:52,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:52,166 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=362 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/B/6b0fa9de8eb242899db3db3d969dff39 2023-04-12 15:54:52,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:52,166 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:52,166 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:52,167 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:52,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:52,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:52,167 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:52,167 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:52,167 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:52,167 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:52,168 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:52,168 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:52,168 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:52,168 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:52,168 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:52,168 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. started... 2023-04-12 15:54:52,168 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. as already flushing 2023-04-12 15:54:52,168 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:52,168 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:52,168 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:52,168 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:52,168 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:52,168 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:52,170 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:52,170 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:52,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:52,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:52,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:52,170 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:52,170 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:52,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:52,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:52,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:52,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:52,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:52,172 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:52,172 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63b8011a[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:52,172 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:52,172 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:52,173 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:52,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:52,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:52,173 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:52,173 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:52,174 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:52,174 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:52,174 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:52,175 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:52,175 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:52,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:52,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:52,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:52,175 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:52,175 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:52,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:52,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:52,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:52,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:52,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:52,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:52,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:52,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:52,178 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=362 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/C/ac2d9286b4ce498988daa32f1d665382 2023-04-12 15:54:52,178 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:52,178 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:52,178 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:52,178 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:52,178 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:52,178 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:52,181 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:52,181 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:52,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:52,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:52,181 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:52,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:52,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:52,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:52,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:52,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:52,182 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:52,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:52,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:52,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:52,182 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/A/1f0c597b4e18471da07895fa82095be9 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/1f0c597b4e18471da07895fa82095be9 2023-04-12 15:54:52,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:52,183 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:52,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:52,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:52,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:52,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:52,186 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:52,186 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:52,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:52,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:52,186 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:52,186 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:52,186 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:52,186 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:52,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:52,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:52,186 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:52,187 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/1f0c597b4e18471da07895fa82095be9, entries=150, sequenceid=362, filesize=11.9 K 2023-04-12 15:54:52,187 DEBUG [Thread-1647] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:52,187 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:52,187 DEBUG [Thread-1647] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:52,187 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:52,187 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:52,187 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:52,187 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:52,187 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:52,188 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/B/6b0fa9de8eb242899db3db3d969dff39 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/6b0fa9de8eb242899db3db3d969dff39 2023-04-12 15:54:52,191 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/6b0fa9de8eb242899db3db3d969dff39, entries=150, sequenceid=362, filesize=11.9 K 2023-04-12 15:54:52,191 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/C/ac2d9286b4ce498988daa32f1d665382 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/ac2d9286b4ce498988daa32f1d665382 2023-04-12 15:54:52,195 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/ac2d9286b4ce498988daa32f1d665382, entries=150, sequenceid=362, filesize=11.9 K 2023-04-12 15:54:52,196 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=120.76 KB/123660 for deda1ce51fef7ef8d26da255318c55e4 in 460ms, sequenceid=362, compaction requested=true 2023-04-12 15:54:52,196 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for deda1ce51fef7ef8d26da255318c55e4: 2023-04-12 15:54:52,196 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:54:52,196 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:54:52,196 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 15:54:52,196 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-12 15:54:52,196 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:54:52,197 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37172 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 15:54:52,197 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1912): deda1ce51fef7ef8d26da255318c55e4/A is initiating minor compaction (all files) 2023-04-12 15:54:52,197 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of deda1ce51fef7ef8d26da255318c55e4/A in TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:52,197 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/901263c6a73b48309269f92b6c0602f5, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/f185d5def0e840c89b7cbab5ec98c899, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/1f0c597b4e18471da07895fa82095be9] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp, totalSize=36.3 K 2023-04-12 15:54:52,198 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49324 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-12 15:54:52,198 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 901263c6a73b48309269f92b6c0602f5, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=338, earliestPutTs=1681314890604 2023-04-12 15:54:52,198 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1912): deda1ce51fef7ef8d26da255318c55e4/B is initiating minor compaction (all files) 2023-04-12 15:54:52,198 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2259): Starting compaction of deda1ce51fef7ef8d26da255318c55e4/B in TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:52,198 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/32c926e62d6f4400b6a6d14109b5b287, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/208bc335880a4650812d288c97195690, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/ca2496e21c8a4e6692f2780a74db43f0, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/6b0fa9de8eb242899db3db3d969dff39] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp, totalSize=48.2 K 2023-04-12 15:54:52,198 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting f185d5def0e840c89b7cbab5ec98c899, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=350, earliestPutTs=1681314890690 2023-04-12 15:54:52,199 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 32c926e62d6f4400b6a6d14109b5b287, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=300, earliestPutTs=1681314890558 2023-04-12 15:54:52,199 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 1f0c597b4e18471da07895fa82095be9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=362, earliestPutTs=1681314891735 2023-04-12 15:54:52,199 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 208bc335880a4650812d288c97195690, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=338, earliestPutTs=1681314890604 2023-04-12 15:54:52,199 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting ca2496e21c8a4e6692f2780a74db43f0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=350, earliestPutTs=1681314890690 2023-04-12 15:54:52,200 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 6b0fa9de8eb242899db3db3d969dff39, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=362, earliestPutTs=1681314891735 2023-04-12 15:54:52,205 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] throttle.PressureAwareThroughputController(145): deda1ce51fef7ef8d26da255318c55e4#A#compaction#264 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:54:52,218 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] throttle.PressureAwareThroughputController(145): deda1ce51fef7ef8d26da255318c55e4#B#compaction#265 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:54:52,244 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/B/5d1b9254dda7434eae6c18809769e973 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/5d1b9254dda7434eae6c18809769e973 2023-04-12 15:54:52,249 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in deda1ce51fef7ef8d26da255318c55e4/B of deda1ce51fef7ef8d26da255318c55e4 into 5d1b9254dda7434eae6c18809769e973(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:54:52,249 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for deda1ce51fef7ef8d26da255318c55e4: 2023-04-12 15:54:52,249 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4., storeName=deda1ce51fef7ef8d26da255318c55e4/B, priority=12, startTime=1681314892196; duration=0sec 2023-04-12 15:54:52,250 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:54:52,250 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-12 15:54:52,251 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49324 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-12 15:54:52,251 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1912): deda1ce51fef7ef8d26da255318c55e4/C is initiating minor compaction (all files) 2023-04-12 15:54:52,251 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2259): Starting compaction of deda1ce51fef7ef8d26da255318c55e4/C in TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:52,251 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/c13908f0ee9943d382d20e68abf2a984, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/1ab3da9470f4491bae562852f0621a4d, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/42507508ca7546ccab2ca456c438dc5f, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/ac2d9286b4ce498988daa32f1d665382] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp, totalSize=48.2 K 2023-04-12 15:54:52,251 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting c13908f0ee9943d382d20e68abf2a984, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=300, earliestPutTs=1681314890558 2023-04-12 15:54:52,252 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 1ab3da9470f4491bae562852f0621a4d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=338, earliestPutTs=1681314890604 2023-04-12 15:54:52,252 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 42507508ca7546ccab2ca456c438dc5f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=350, earliestPutTs=1681314890690 2023-04-12 15:54:52,253 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting ac2d9286b4ce498988daa32f1d665382, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=362, earliestPutTs=1681314891735 2023-04-12 15:54:52,261 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] throttle.PressureAwareThroughputController(145): deda1ce51fef7ef8d26da255318c55e4#C#compaction#266 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:54:52,269 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/C/3f74406dcb054cd9a061f91be65afd11 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/3f74406dcb054cd9a061f91be65afd11 2023-04-12 15:54:52,275 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in deda1ce51fef7ef8d26da255318c55e4/C of deda1ce51fef7ef8d26da255318c55e4 into 3f74406dcb054cd9a061f91be65afd11(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:54:52,275 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for deda1ce51fef7ef8d26da255318c55e4: 2023-04-12 15:54:52,275 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4., storeName=deda1ce51fef7ef8d26da255318c55e4/C, priority=12, startTime=1681314892196; duration=0sec 2023-04-12 15:54:52,275 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:54:52,287 DEBUG [Thread-1647] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:52,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:52,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:52,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:52,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:52,290 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:52,290 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:52,290 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:52,290 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:52,291 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:52,291 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:52,292 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:52,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:52,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:52,292 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:52,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:52,292 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:52,292 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:52,292 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:52,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:52,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:52,293 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:52,293 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:52,293 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:52,293 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:52,293 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:52,293 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:52,293 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:52,294 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:52,294 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:52,294 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. started... 2023-04-12 15:54:52,294 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing deda1ce51fef7ef8d26da255318c55e4 3/3 column families, dataSize=120.76 KB heapSize=340.45 KB 2023-04-12 15:54:52,298 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK deda1ce51fef7ef8d26da255318c55e4, store=A 2023-04-12 15:54:52,298 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:54:52,298 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK deda1ce51fef7ef8d26da255318c55e4, store=B 2023-04-12 15:54:52,298 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:54:52,298 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK deda1ce51fef7ef8d26da255318c55e4, store=C 2023-04-12 15:54:52,298 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:54:52,314 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=398 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/A/0aa0cf31ecfb4f3989e97428c9edfde9 2023-04-12 15:54:52,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on deda1ce51fef7ef8d26da255318c55e4 2023-04-12 15:54:52,398 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. as already flushing 2023-04-12 15:54:52,427 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:52,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53040 deadline: 1681314952425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:52,429 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:52,429 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:52,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53012 deadline: 1681314952425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:52,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53078 deadline: 1681314952426, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:52,532 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:52,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53012 deadline: 1681314952530, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:52,634 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/A/5a4f7d456d4748ec8a5c97c6a3ddcfec as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/5a4f7d456d4748ec8a5c97c6a3ddcfec 2023-04-12 15:54:52,639 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in deda1ce51fef7ef8d26da255318c55e4/A of deda1ce51fef7ef8d26da255318c55e4 into 5a4f7d456d4748ec8a5c97c6a3ddcfec(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:54:52,639 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for deda1ce51fef7ef8d26da255318c55e4: 2023-04-12 15:54:52,639 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4., storeName=deda1ce51fef7ef8d26da255318c55e4/A, priority=13, startTime=1681314892196; duration=0sec 2023-04-12 15:54:52,640 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:54:52,735 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:52,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53012 deadline: 1681314952734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:52,736 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=398 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/B/d4a7809cb7e64e7da0639b8aeca65e7c 2023-04-12 15:54:52,824 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:52,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53032 deadline: 1681314952823, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:52,828 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:52,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53014 deadline: 1681314952827, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:52,931 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:52,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53078 deadline: 1681314952930, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:52,934 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:52,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53040 deadline: 1681314952932, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:53,055 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:53,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53012 deadline: 1681314953052, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:53,150 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=398 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/C/37aa5784eb6048898414c6a5fa66dd7a 2023-04-12 15:54:53,155 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/A/0aa0cf31ecfb4f3989e97428c9edfde9 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/0aa0cf31ecfb4f3989e97428c9edfde9 2023-04-12 15:54:53,159 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/0aa0cf31ecfb4f3989e97428c9edfde9, entries=150, sequenceid=398, filesize=11.9 K 2023-04-12 15:54:53,163 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/B/d4a7809cb7e64e7da0639b8aeca65e7c as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/d4a7809cb7e64e7da0639b8aeca65e7c 2023-04-12 15:54:53,168 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/d4a7809cb7e64e7da0639b8aeca65e7c, entries=150, sequenceid=398, filesize=11.9 K 2023-04-12 15:54:53,169 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/C/37aa5784eb6048898414c6a5fa66dd7a as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/37aa5784eb6048898414c6a5fa66dd7a 2023-04-12 15:54:53,172 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/37aa5784eb6048898414c6a5fa66dd7a, entries=150, sequenceid=398, filesize=11.9 K 2023-04-12 15:54:53,173 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~340.41 KB/348576, currentSize=60.38 KB/61830 for deda1ce51fef7ef8d26da255318c55e4 in 879ms, sequenceid=398, compaction requested=false 2023-04-12 15:54:53,173 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for deda1ce51fef7ef8d26da255318c55e4: 2023-04-12 15:54:53,173 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:53,173 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:53,173 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:53,173 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:53,173 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:53,174 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:53,182 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:53,182 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:53,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:53,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:53,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:53,182 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:53,182 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:53,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:53,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:53,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:53,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:53,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:53,184 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:53,184 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5161f298[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:53,184 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:53,184 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:53,185 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:53,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:53,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:53,185 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:53,185 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:53,185 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:53,185 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:53,185 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:53,187 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:53,187 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:53,187 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:53,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:53,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:53,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:53,187 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:53,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:53,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:53,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:53,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:53,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:53,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:53,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:53,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:53,190 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:53,190 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:53,190 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:53,190 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:53,190 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:53,190 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:53,191 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:53,191 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:53,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:53,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:53,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:53,191 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:53,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:53,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:53,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:53,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:53,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:53,192 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:53,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:53,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:53,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:53,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:53,193 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:53,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:53,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:53,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:53,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:53,197 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:53,197 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:53,197 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:53,197 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:53,197 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:53,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:53,197 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:53,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:53,197 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:53,197 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:53,197 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:53,197 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:53,197 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:53,197 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:53,197 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:53,197 DEBUG [Thread-1647] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:53,197 DEBUG [Thread-1647] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:53,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:53,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:53,297 DEBUG [Thread-1647] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:53,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:53,300 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:53,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:53,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:53,301 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:53,301 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:53,301 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:53,301 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:53,301 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:53,301 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:53,303 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:53,303 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:53,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:53,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:53,303 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:53,303 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:53,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:53,303 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:53,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:53,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:53,304 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:53,304 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:53,304 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:53,304 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:53,304 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:53,304 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:53,307 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:53,307 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:53,307 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:53,308 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. started... 2023-04-12 15:54:53,308 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing deda1ce51fef7ef8d26da255318c55e4 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-12 15:54:53,308 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK deda1ce51fef7ef8d26da255318c55e4, store=A 2023-04-12 15:54:53,308 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:53,308 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK deda1ce51fef7ef8d26da255318c55e4, store=B 2023-04-12 15:54:53,308 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:53,308 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK deda1ce51fef7ef8d26da255318c55e4, store=C 2023-04-12 15:54:53,308 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:53,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on deda1ce51fef7ef8d26da255318c55e4 2023-04-12 15:54:53,559 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. as already flushing 2023-04-12 15:54:53,603 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:53,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53012 deadline: 1681314953602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:53,706 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:53,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53012 deadline: 1681314953704, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:53,723 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=411 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/A/7082019db147440fa332ddaada47ebd7 2023-04-12 15:54:53,737 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=411 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/B/7603c8a2eb4a4341b21b81aba27eea17 2023-04-12 15:54:53,909 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:53,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53012 deadline: 1681314953908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:53,935 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:53,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53040 deadline: 1681314953935, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:53,943 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:53,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53078 deadline: 1681314953941, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,147 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=411 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/C/acf4b06b154f4cd0be674aaa7727df3c 2023-04-12 15:54:54,151 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/A/7082019db147440fa332ddaada47ebd7 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/7082019db147440fa332ddaada47ebd7 2023-04-12 15:54:54,155 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/7082019db147440fa332ddaada47ebd7, entries=150, sequenceid=411, filesize=11.9 K 2023-04-12 15:54:54,155 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/B/7603c8a2eb4a4341b21b81aba27eea17 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/7603c8a2eb4a4341b21b81aba27eea17 2023-04-12 15:54:54,160 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/7603c8a2eb4a4341b21b81aba27eea17, entries=150, sequenceid=411, filesize=11.9 K 2023-04-12 15:54:54,161 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/C/acf4b06b154f4cd0be674aaa7727df3c as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/acf4b06b154f4cd0be674aaa7727df3c 2023-04-12 15:54:54,164 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/acf4b06b154f4cd0be674aaa7727df3c, entries=150, sequenceid=411, filesize=11.9 K 2023-04-12 15:54:54,164 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for deda1ce51fef7ef8d26da255318c55e4 in 856ms, sequenceid=411, compaction requested=true 2023-04-12 15:54:54,164 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for deda1ce51fef7ef8d26da255318c55e4: 2023-04-12 15:54:54,165 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:54,165 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:54,165 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:54,165 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,165 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:54,165 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:54,166 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,166 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:54,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:54,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:54,166 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:54,167 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:54,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:54,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:54,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:54,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:54,168 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:54,168 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38e15756[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:54,168 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:54,168 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:54,169 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:54,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:54,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:54,169 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:54,169 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:54,169 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:54,170 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,170 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:54,170 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:54,170 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:54,170 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,171 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:54,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:54,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:54,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:54,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:54,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:54,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:54,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:54,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:54,173 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:54,173 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:54,173 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:54,173 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:54,173 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:54,173 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:54,176 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:54,176 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:54,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:54,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:54,176 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:54,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:54,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:54,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:54,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:54,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:54,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:54,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:54,177 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:54,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:54,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:54,178 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:54,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:54,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,180 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:54,180 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:54,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:54,180 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:54,180 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:54,180 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:54,180 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:54,181 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:54,181 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:54,181 DEBUG [Thread-1647] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:54,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:54,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:54,181 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,181 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:54,181 DEBUG [Thread-1647] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:54,181 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:54,181 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:54,181 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:54,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on deda1ce51fef7ef8d26da255318c55e4 2023-04-12 15:54:54,212 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing deda1ce51fef7ef8d26da255318c55e4 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-12 15:54:54,212 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK deda1ce51fef7ef8d26da255318c55e4, store=A 2023-04-12 15:54:54,213 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:54:54,213 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK deda1ce51fef7ef8d26da255318c55e4, store=B 2023-04-12 15:54:54,213 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:54:54,213 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK deda1ce51fef7ef8d26da255318c55e4, store=C 2023-04-12 15:54:54,213 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:54:54,227 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=432 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/A/0010282370924d608bbf17e6606abc93 2023-04-12 15:54:54,239 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:54,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53012 deadline: 1681314954236, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,281 DEBUG [Thread-1647] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:54,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:54,283 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:54,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:54,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:54,284 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:54,284 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:54,284 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:54,284 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:54,285 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:54,285 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:54,286 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,286 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:54,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:54,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:54,286 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,286 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:54,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:54,287 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:54,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:54,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:54,287 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:54,287 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:54,287 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:54,287 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:54,287 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:54,287 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:54,288 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:54,288 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:54,288 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:54,288 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. started... 2023-04-12 15:54:54,288 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. as already flushing 2023-04-12 15:54:54,288 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:54,288 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:54,288 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:54,288 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,289 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:54,289 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:54,290 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,290 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:54,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:54,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:54,290 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:54,290 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:54,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:54,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:54,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:54,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:54,292 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:54,292 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@117577f6[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:54,292 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:54,292 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:54,294 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:54,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:54,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:54,294 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:54,294 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:54,294 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:54,294 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,294 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:54,296 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,296 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:54,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:54,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:54,296 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:54,296 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:54,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:54,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:54,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:54,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:54,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:54,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:54,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:54,298 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:54,298 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:54,298 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:54,298 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:54,298 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:54,300 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:54,300 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:54,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:54,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:54,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:54,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:54,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:54,300 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:54,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:54,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:54,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:54,301 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:54,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:54,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:54,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:54,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,302 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:54,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:54,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,305 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,305 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:54,305 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:54,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:54,305 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:54,305 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:54,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:54,305 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:54,305 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:54,305 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:54,305 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,305 DEBUG [Thread-1647] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:54,305 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:54,305 DEBUG [Thread-1647] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:54,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:54,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:54,305 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:54,305 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:54,342 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:54,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53012 deadline: 1681314954340, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,405 DEBUG [Thread-1647] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:54,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:54,407 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:54,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:54,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:54,408 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:54,408 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:54,408 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:54,408 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:54,409 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:54,409 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:54,411 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,411 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:54,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:54,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:54,411 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,411 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:54,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:54,411 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:54,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:54,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:54,412 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:54,412 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:54,412 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:54,412 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:54,412 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:54,412 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:54,412 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:54,413 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:54,413 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:54,413 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. started... 2023-04-12 15:54:54,413 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. as already flushing 2023-04-12 15:54:54,413 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:54,413 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:54,413 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:54,413 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,413 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:54,413 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:54,414 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,414 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:54,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:54,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:54,415 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:54,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:54,415 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:54,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:54,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:54,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:54,416 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:54,416 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32e60413[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:54,416 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:54,416 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:54,417 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:54,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:54,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:54,417 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:54,417 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:54,417 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:54,417 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,417 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:54,421 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,421 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:54,422 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:54,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:54,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:54,422 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:54,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:54,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:54,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:54,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:54,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:54,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:54,424 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:54,424 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:54,424 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:54,424 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:54,424 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:54,424 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:54,426 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:54,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:54,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:54,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:54,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:54,427 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:54,427 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:54,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:54,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:54,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:54,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:54,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:54,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:54,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:54,428 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:54,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:54,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:54,429 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,431 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,431 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:54,431 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:54,431 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,431 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:54,431 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:54,431 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:54,431 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:54,431 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:54,431 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:54,432 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:54,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:54,431 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:54,432 DEBUG [Thread-1647] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:54,432 DEBUG [Thread-1647] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:54,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:54,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:54,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:54,532 DEBUG [Thread-1647] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:54,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:54,534 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:54,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:54,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:54,535 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:54,535 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:54,535 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:54,535 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:54,535 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:54,536 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:54,537 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:54,537 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:54,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:54,537 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,537 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:54,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:54,538 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:54,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:54,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:54,538 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:54,538 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:54,538 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:54,538 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:54,538 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:54,539 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:54,539 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:54,539 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:54,539 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:54,539 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. started... 2023-04-12 15:54:54,539 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. as already flushing 2023-04-12 15:54:54,539 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:54,539 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:54,539 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:54,539 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,539 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:54,539 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:54,542 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:54,542 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:54,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:54,542 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:54,542 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:54,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:54,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:54,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:54,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:54,544 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:54,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53012 deadline: 1681314954543, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,544 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:54,544 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c4dd5e5[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:54,544 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:54,544 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:54,545 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:54,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:54,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:54,545 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:54,545 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,546 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:54,545 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:54,546 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:54,547 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:54,547 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,547 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:54,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:54,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:54,547 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:54,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:54,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:54,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:54,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:54,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:54,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:54,549 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:54,549 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:54,549 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:54,549 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:54,550 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:54,550 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:54,551 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:54,551 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:54,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:54,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:54,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:54,551 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:54,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:54,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:54,551 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:54,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:54,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:54,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:54,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:54,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:54,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:54,553 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:54,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:54,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,556 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,556 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:54,556 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:54,556 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:54,556 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:54,556 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:54,556 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:54,556 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:54,556 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:54,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:54,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:54,556 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,557 DEBUG [Thread-1647] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:54,557 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:54,557 DEBUG [Thread-1647] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:54,557 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:54,557 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:54,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:54,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:54,638 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=432 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/B/e242b7a5ebf7415e9661fc10574b4075 2023-04-12 15:54:54,657 DEBUG [Thread-1647] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:54,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:54,659 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:54,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:54,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:54,660 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:54,660 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:54,660 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:54,660 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:54,660 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:54,660 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:54,661 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,661 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:54,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:54,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:54,662 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,662 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:54,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:54,662 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:54,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:54,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:54,662 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:54,663 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:54,663 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:54,663 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:54,663 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:54,663 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:54,663 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:54,663 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:54,663 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:54,663 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. started... 2023-04-12 15:54:54,663 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. as already flushing 2023-04-12 15:54:54,663 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:54,663 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:54,663 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:54,663 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,663 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:54,663 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:54,665 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,665 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:54,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:54,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:54,665 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:54,665 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:54,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:54,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:54,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:54,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:54,666 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:54,666 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ea474dd[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:54,666 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:54,666 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:54,667 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:54,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:54,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:54,667 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:54,667 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:54,667 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:54,667 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,668 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:54,670 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,670 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:54,670 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:54,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:54,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:54,670 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:54,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:54,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:54,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:54,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:54,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:54,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:54,673 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:54,673 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:54,673 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:54,673 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:54,673 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:54,673 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:54,674 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:54,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:54,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:54,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:54,674 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:54,674 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:54,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:54,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:54,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:54,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:54,675 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:54,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:54,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:54,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:54,676 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:54,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:54,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:54,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,678 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,678 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:54,678 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:54,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:54,678 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:54,678 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:54,678 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:54,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:54,678 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:54,678 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,678 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:54,678 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:54,678 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:54,678 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:54,678 DEBUG [Thread-1647] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:54,678 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:54,678 DEBUG [Thread-1647] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:54,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:54,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:54,779 DEBUG [Thread-1647] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:54,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:54,781 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:54,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:54,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:54,782 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:54,782 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:54,782 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:54,782 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:54,783 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:54,783 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:54,784 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,784 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:54,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:54,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:54,784 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,784 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:54,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:54,785 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:54,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:54,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:54,785 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:54,785 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:54,785 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:54,785 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:54,785 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:54,785 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:54,785 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:54,786 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:54,786 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:54,786 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. started... 2023-04-12 15:54:54,786 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. as already flushing 2023-04-12 15:54:54,786 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:54,786 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:54,786 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:54,786 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,786 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:54,786 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:54,788 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,788 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:54,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:54,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:54,789 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:54,789 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:54,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:54,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:54,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:54,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:54,791 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:54,791 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64547238[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:54,791 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:54,791 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:54,792 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:54,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:54,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:54,792 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:54,792 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:54,792 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:54,792 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,792 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:54,793 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,793 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:54,794 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:54,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:54,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:54,794 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:54,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:54,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:54,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:54,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:54,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:54,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:54,796 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:54,796 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:54,796 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:54,796 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:54,796 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:54,796 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:54,801 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:54,801 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:54,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:54,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:54,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:54,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:54,801 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:54,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:54,801 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:54,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:54,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:54,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:54,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:54,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:54,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:54,803 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:54,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:54,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,805 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:54,805 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:54,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:54,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:54,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:54,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:54,805 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,805 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:54,805 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:54,805 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,805 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:54,805 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:54,805 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:54,805 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:54,805 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:54,805 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:54,806 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:54,806 DEBUG [Thread-1647] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:54,806 DEBUG [Thread-1647] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:54,836 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:54,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53032 deadline: 1681314954834, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,837 DEBUG [Thread-1641] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4138 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4., hostname=jenkins-hbase4.apache.org,38905,1681314818739, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-12 15:54:54,843 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:54,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53014 deadline: 1681314954841, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,845 DEBUG [Thread-1637] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4147 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4., hostname=jenkins-hbase4.apache.org,38905,1681314818739, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-12 15:54:54,850 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:54,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53012 deadline: 1681314954847, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,906 DEBUG [Thread-1647] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:54,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:54,908 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:54,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:54,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:54,909 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:54,909 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:54,909 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:54,909 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:54,909 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:54,910 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:54,911 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,911 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:54,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:54,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:54,911 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,911 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:54,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:54,911 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:54,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:54,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:54,912 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:54,912 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:54,912 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:54,912 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:54,912 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:54,912 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:54,912 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:54,912 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:54,912 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:54,912 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. started... 2023-04-12 15:54:54,912 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. as already flushing 2023-04-12 15:54:54,912 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:54,912 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:54,913 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:54,913 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,913 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:54,913 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:54,914 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:54,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:54,914 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:54,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:54,914 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:54,914 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:54,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:54,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:54,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:54,915 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:54,915 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6610243d[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:54,915 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:54,916 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:54,917 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:54,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:54,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:54,917 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:54,918 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:54,918 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,918 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:54,918 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:54,919 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,919 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:54,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:54,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:54,919 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:54,919 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:54,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:54,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:54,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:54,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:54,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:54,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:54,921 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:54,921 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:54,921 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:54,921 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:54,921 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:54,921 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:54,922 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:54,922 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:54,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:54,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:54,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:54,923 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:54,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:54,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:54,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:54,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:54,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:54,923 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:54,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:54,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:54,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:54,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,924 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:54,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:54,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,926 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,926 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:54,926 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:54,927 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:54,926 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:54,927 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:54,927 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:54,927 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:54,927 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:54,927 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:54,927 DEBUG [Thread-1647] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:54,927 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:54,927 DEBUG [Thread-1647] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:54,927 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:54,927 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:54,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:54,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:54,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:54,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:55,027 DEBUG [Thread-1647] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:55,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:55,029 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:55,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:55,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:55,030 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:55,031 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:55,031 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:55,031 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:55,031 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:55,031 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:55,033 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:55,033 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:55,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:55,033 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:55,033 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:55,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:55,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:55,034 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:55,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:55,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:55,034 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:55,035 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:55,035 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:55,035 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:55,035 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:55,035 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:55,035 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:55,035 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:55,035 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:55,035 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. started... 2023-04-12 15:54:55,036 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. as already flushing 2023-04-12 15:54:55,036 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:55,036 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:55,036 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:55,036 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:55,036 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:55,036 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:55,037 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:55,037 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:55,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:55,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:55,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:55,038 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:55,038 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:55,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:55,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:55,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:55,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:55,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:55,039 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:55,039 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@647027e7[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:55,039 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:55,039 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:55,040 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:55,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:55,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:55,040 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:55,040 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:55,040 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:55,041 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:55,041 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:55,041 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:55,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:55,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:55,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:55,042 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:55,042 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:55,042 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:55,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:55,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:55,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:55,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:55,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:55,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:55,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:55,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:55,044 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:55,044 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:55,044 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:55,044 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:55,044 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:55,044 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:55,047 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:55,047 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:55,047 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:55,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:55,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:55,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:55,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:55,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:55,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:55,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:55,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:55,047 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:55,048 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=432 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/C/fa57eaec78f4480ab427de0528ab895d 2023-04-12 15:54:55,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:55,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:55,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:55,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:55,049 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:55,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:55,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:55,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:55,052 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:55,052 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:55,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:55,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:55,052 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:55,052 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:55,052 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:55,052 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:55,052 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:55,052 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:55,052 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:55,053 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:55,052 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:55,053 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:55,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:55,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:55,053 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/A/0010282370924d608bbf17e6606abc93 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/0010282370924d608bbf17e6606abc93 2023-04-12 15:54:55,053 DEBUG [Thread-1647] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:55,053 DEBUG [Thread-1647] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:55,054 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:55,057 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/0010282370924d608bbf17e6606abc93, entries=200, sequenceid=432, filesize=14.3 K 2023-04-12 15:54:55,057 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/B/e242b7a5ebf7415e9661fc10574b4075 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/e242b7a5ebf7415e9661fc10574b4075 2023-04-12 15:54:55,061 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/e242b7a5ebf7415e9661fc10574b4075, entries=150, sequenceid=432, filesize=11.9 K 2023-04-12 15:54:55,061 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/C/fa57eaec78f4480ab427de0528ab895d as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/fa57eaec78f4480ab427de0528ab895d 2023-04-12 15:54:55,065 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/fa57eaec78f4480ab427de0528ab895d, entries=150, sequenceid=432, filesize=11.9 K 2023-04-12 15:54:55,065 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=53.67 KB/54960 for deda1ce51fef7ef8d26da255318c55e4 in 853ms, sequenceid=432, compaction requested=true 2023-04-12 15:54:55,065 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for deda1ce51fef7ef8d26da255318c55e4: 2023-04-12 15:54:55,066 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:54:55,066 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-12 15:54:55,066 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:54:55,066 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:54:55,066 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-12 15:54:55,067 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 51900 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-12 15:54:55,067 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1912): deda1ce51fef7ef8d26da255318c55e4/A is initiating minor compaction (all files) 2023-04-12 15:54:55,067 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of deda1ce51fef7ef8d26da255318c55e4/A in TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:55,067 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/5a4f7d456d4748ec8a5c97c6a3ddcfec, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/0aa0cf31ecfb4f3989e97428c9edfde9, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/7082019db147440fa332ddaada47ebd7, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/0010282370924d608bbf17e6606abc93] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp, totalSize=50.7 K 2023-04-12 15:54:55,068 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49460 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-12 15:54:55,068 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1912): deda1ce51fef7ef8d26da255318c55e4/B is initiating minor compaction (all files) 2023-04-12 15:54:55,068 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2259): Starting compaction of deda1ce51fef7ef8d26da255318c55e4/B in TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:55,068 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 5a4f7d456d4748ec8a5c97c6a3ddcfec, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=362, earliestPutTs=1681314891735 2023-04-12 15:54:55,068 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/5d1b9254dda7434eae6c18809769e973, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/d4a7809cb7e64e7da0639b8aeca65e7c, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/7603c8a2eb4a4341b21b81aba27eea17, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/e242b7a5ebf7415e9661fc10574b4075] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp, totalSize=48.3 K 2023-04-12 15:54:55,069 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 0aa0cf31ecfb4f3989e97428c9edfde9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=398, earliestPutTs=1681314891808 2023-04-12 15:54:55,069 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 5d1b9254dda7434eae6c18809769e973, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=362, earliestPutTs=1681314891735 2023-04-12 15:54:55,069 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 7082019db147440fa332ddaada47ebd7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=411, earliestPutTs=1681314892401 2023-04-12 15:54:55,069 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting d4a7809cb7e64e7da0639b8aeca65e7c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=398, earliestPutTs=1681314891808 2023-04-12 15:54:55,069 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 0010282370924d608bbf17e6606abc93, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=432, earliestPutTs=1681314893586 2023-04-12 15:54:55,069 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 7603c8a2eb4a4341b21b81aba27eea17, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=411, earliestPutTs=1681314892401 2023-04-12 15:54:55,070 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting e242b7a5ebf7415e9661fc10574b4075, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=432, earliestPutTs=1681314893586 2023-04-12 15:54:55,079 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] throttle.PressureAwareThroughputController(145): deda1ce51fef7ef8d26da255318c55e4#A#compaction#276 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:54:55,080 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] throttle.PressureAwareThroughputController(145): deda1ce51fef7ef8d26da255318c55e4#B#compaction#277 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:54:55,093 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/B/65fb7b97edba4ca88b4b69faf2b89d73 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/65fb7b97edba4ca88b4b69faf2b89d73 2023-04-12 15:54:55,097 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in deda1ce51fef7ef8d26da255318c55e4/B of deda1ce51fef7ef8d26da255318c55e4 into 65fb7b97edba4ca88b4b69faf2b89d73(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:54:55,097 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for deda1ce51fef7ef8d26da255318c55e4: 2023-04-12 15:54:55,097 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4., storeName=deda1ce51fef7ef8d26da255318c55e4/B, priority=12, startTime=1681314895066; duration=0sec 2023-04-12 15:54:55,097 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:54:55,097 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-12 15:54:55,098 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49460 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-12 15:54:55,098 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1912): deda1ce51fef7ef8d26da255318c55e4/C is initiating minor compaction (all files) 2023-04-12 15:54:55,099 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2259): Starting compaction of deda1ce51fef7ef8d26da255318c55e4/C in TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:55,099 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/3f74406dcb054cd9a061f91be65afd11, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/37aa5784eb6048898414c6a5fa66dd7a, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/acf4b06b154f4cd0be674aaa7727df3c, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/fa57eaec78f4480ab427de0528ab895d] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp, totalSize=48.3 K 2023-04-12 15:54:55,099 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 3f74406dcb054cd9a061f91be65afd11, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=362, earliestPutTs=1681314891735 2023-04-12 15:54:55,099 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 37aa5784eb6048898414c6a5fa66dd7a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=398, earliestPutTs=1681314891808 2023-04-12 15:54:55,100 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting acf4b06b154f4cd0be674aaa7727df3c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=411, earliestPutTs=1681314892401 2023-04-12 15:54:55,100 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting fa57eaec78f4480ab427de0528ab895d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=432, earliestPutTs=1681314893586 2023-04-12 15:54:55,106 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] throttle.PressureAwareThroughputController(145): deda1ce51fef7ef8d26da255318c55e4#C#compaction#278 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:54:55,153 DEBUG [Thread-1647] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:55,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:55,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:55,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:55,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:55,156 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:55,156 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:55,156 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:55,156 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:55,157 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:55,157 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:55,158 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:55,158 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:55,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:55,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:55,158 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:55,158 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:55,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:55,159 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:55,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:55,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:55,159 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:55,159 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:55,159 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:55,159 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:55,159 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:55,159 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:55,160 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:55,160 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:55,160 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:55,160 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. started... 2023-04-12 15:54:55,160 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing deda1ce51fef7ef8d26da255318c55e4 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-12 15:54:55,160 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK deda1ce51fef7ef8d26da255318c55e4, store=A 2023-04-12 15:54:55,160 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:55,160 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK deda1ce51fef7ef8d26da255318c55e4, store=B 2023-04-12 15:54:55,160 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:55,160 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK deda1ce51fef7ef8d26da255318c55e4, store=C 2023-04-12 15:54:55,160 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:55,170 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=444 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/A/986287c66bc8405a8f5a2cf776e27c2b 2023-04-12 15:54:55,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on deda1ce51fef7ef8d26da255318c55e4 2023-04-12 15:54:55,353 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. as already flushing 2023-04-12 15:54:55,398 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681314895384, max timestamp=1681314895396Num uniques 859; 2023-04-12 15:54:55,398 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681314895384, max timestamp=1681314895396Num uniques 859; 2023-04-12 15:54:55,399 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-12 15:54:55,398 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681314895384, max timestamp=1681314895396Num uniques 859; 2023-04-12 15:54:55,398 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-12 15:54:55,399 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-12 15:54:55,446 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:55,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53012 deadline: 1681314955443, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:55,490 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/A/b76e0fd08e50492eb7b5afe7b85fa74c as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/b76e0fd08e50492eb7b5afe7b85fa74c 2023-04-12 15:54:55,494 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in deda1ce51fef7ef8d26da255318c55e4/A of deda1ce51fef7ef8d26da255318c55e4 into b76e0fd08e50492eb7b5afe7b85fa74c(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:54:55,494 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for deda1ce51fef7ef8d26da255318c55e4: 2023-04-12 15:54:55,494 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4., storeName=deda1ce51fef7ef8d26da255318c55e4/A, priority=12, startTime=1681314895066; duration=0sec 2023-04-12 15:54:55,494 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:54:55,515 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/C/fd9beb5e5eea428e847978b6ba355347 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/fd9beb5e5eea428e847978b6ba355347 2023-04-12 15:54:55,519 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in deda1ce51fef7ef8d26da255318c55e4/C of deda1ce51fef7ef8d26da255318c55e4 into fd9beb5e5eea428e847978b6ba355347(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:54:55,519 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for deda1ce51fef7ef8d26da255318c55e4: 2023-04-12 15:54:55,519 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4., storeName=deda1ce51fef7ef8d26da255318c55e4/C, priority=12, startTime=1681314895066; duration=0sec 2023-04-12 15:54:55,519 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:54:55,548 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:55,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53012 deadline: 1681314955547, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:55,579 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=444 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/B/6f13f9cc00884607abb5418d7a5c234c 2023-04-12 15:54:55,593 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=444 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/C/45afff1641b64118a1aefd2efb4c0a20 2023-04-12 15:54:55,596 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/A/986287c66bc8405a8f5a2cf776e27c2b as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/986287c66bc8405a8f5a2cf776e27c2b 2023-04-12 15:54:55,600 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/986287c66bc8405a8f5a2cf776e27c2b, entries=100, sequenceid=444, filesize=9.5 K 2023-04-12 15:54:55,600 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/B/6f13f9cc00884607abb5418d7a5c234c as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/6f13f9cc00884607abb5418d7a5c234c 2023-04-12 15:54:55,604 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/6f13f9cc00884607abb5418d7a5c234c, entries=100, sequenceid=444, filesize=9.5 K 2023-04-12 15:54:55,605 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/C/45afff1641b64118a1aefd2efb4c0a20 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/45afff1641b64118a1aefd2efb4c0a20 2023-04-12 15:54:55,608 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/45afff1641b64118a1aefd2efb4c0a20, entries=100, sequenceid=444, filesize=9.5 K 2023-04-12 15:54:55,609 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for deda1ce51fef7ef8d26da255318c55e4 in 449ms, sequenceid=444, compaction requested=false 2023-04-12 15:54:55,609 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for deda1ce51fef7ef8d26da255318c55e4: 2023-04-12 15:54:55,609 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:55,609 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:55,609 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:55,609 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:55,612 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:55,612 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:55,613 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:55,613 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:55,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:55,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:55,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:55,614 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:55,614 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:55,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:55,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:55,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:55,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:55,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:55,616 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:55,616 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@190fb528[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:55,616 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:55,616 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:55,618 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:55,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:55,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:55,618 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:55,618 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:55,618 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:55,618 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:55,618 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:55,619 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:55,619 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:55,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:55,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:55,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:55,619 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:55,620 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:55,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:55,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:55,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:55,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:55,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:55,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:55,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:55,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:55,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:55,622 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:55,622 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:55,622 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:55,622 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:55,622 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:55,623 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:55,623 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:55,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:55,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:55,623 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:55,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:55,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:55,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:55,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:55,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:55,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:55,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:55,624 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:55,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:55,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:55,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:55,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:55,625 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:55,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:55,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:55,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:55,630 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:55,630 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:55,630 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:55,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:55,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:55,631 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:55,631 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:55,630 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:55,630 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:55,631 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:55,631 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:55,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:55,631 DEBUG [Thread-1647] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:55,631 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:55,631 DEBUG [Thread-1647] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:55,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:55,631 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:55,631 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:55,731 DEBUG [Thread-1647] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:55,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:55,733 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:55,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:55,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:55,734 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:55,734 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:55,734 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:55,734 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:55,735 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:55,735 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:55,736 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:55,736 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:55,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:55,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:55,736 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:55,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:55,736 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:55,737 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:55,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:55,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:55,737 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:55,737 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:55,737 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:55,737 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:55,737 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:55,737 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:55,738 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:55,738 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:55,738 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:55,738 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. started... 2023-04-12 15:54:55,738 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing deda1ce51fef7ef8d26da255318c55e4 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-12 15:54:55,738 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK deda1ce51fef7ef8d26da255318c55e4, store=A 2023-04-12 15:54:55,738 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:54:55,738 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK deda1ce51fef7ef8d26da255318c55e4, store=B 2023-04-12 15:54:55,738 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:54:55,738 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK deda1ce51fef7ef8d26da255318c55e4, store=C 2023-04-12 15:54:55,738 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:54:55,749 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=482 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/A/69060ab698c841fab0c5f63b3b261569 2023-04-12 15:54:55,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on deda1ce51fef7ef8d26da255318c55e4 2023-04-12 15:54:55,752 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. as already flushing 2023-04-12 15:54:55,782 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:55,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53012 deadline: 1681314955779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:55,883 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:55,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53012 deadline: 1681314955883, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:55,947 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:55,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53078 deadline: 1681314955947, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:55,949 DEBUG [Thread-1639] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4135 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4., hostname=jenkins-hbase4.apache.org,38905,1681314818739, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-12 15:54:55,952 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:55,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53040 deadline: 1681314955949, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:55,953 DEBUG [Thread-1643] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4139 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4., hostname=jenkins-hbase4.apache.org,38905,1681314818739, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-12 15:54:56,086 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:56,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53012 deadline: 1681314956084, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:56,161 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=482 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/B/d978b91e475f436db70e31596c9b65d9 2023-04-12 15:54:56,390 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:56,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53012 deadline: 1681314956388, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:56,571 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=482 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/C/1d647b39904642e683f14de2ef3b913b 2023-04-12 15:54:56,575 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/A/69060ab698c841fab0c5f63b3b261569 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/69060ab698c841fab0c5f63b3b261569 2023-04-12 15:54:56,579 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/69060ab698c841fab0c5f63b3b261569, entries=150, sequenceid=482, filesize=11.9 K 2023-04-12 15:54:56,580 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/B/d978b91e475f436db70e31596c9b65d9 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/d978b91e475f436db70e31596c9b65d9 2023-04-12 15:54:56,583 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/d978b91e475f436db70e31596c9b65d9, entries=150, sequenceid=482, filesize=11.9 K 2023-04-12 15:54:56,584 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/C/1d647b39904642e683f14de2ef3b913b as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/1d647b39904642e683f14de2ef3b913b 2023-04-12 15:54:56,588 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/1d647b39904642e683f14de2ef3b913b, entries=150, sequenceid=482, filesize=11.9 K 2023-04-12 15:54:56,588 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for deda1ce51fef7ef8d26da255318c55e4 in 850ms, sequenceid=482, compaction requested=true 2023-04-12 15:54:56,588 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for deda1ce51fef7ef8d26da255318c55e4: 2023-04-12 15:54:56,588 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:56,588 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:56,589 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:56,589 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:56,589 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:56,589 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:56,590 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:56,590 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:56,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:56,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:56,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:56,590 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:56,590 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:56,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:56,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:56,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:56,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:56,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:56,592 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:56,592 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e0dc183[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:56,592 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:56,592 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:56,593 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:56,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:56,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:56,593 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:56,593 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:56,593 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:56,593 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:56,593 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:56,595 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:56,596 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:56,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:56,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:56,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:56,596 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:56,596 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:56,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:56,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:56,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:56,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:56,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:56,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:56,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:56,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:56,598 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:56,598 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:56,598 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:56,598 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:56,598 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:56,598 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:56,600 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:56,600 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:56,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:56,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:56,600 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:56,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:56,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:56,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:56,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:56,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:56,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:56,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:56,601 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:56,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:56,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:56,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:56,602 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:56,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:56,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:56,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:56,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:56,604 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:56,604 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:56,604 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:56,605 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:56,605 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:56,604 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:56,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:56,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:56,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:56,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:56,605 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:56,605 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:56,605 DEBUG [Thread-1647] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:56,605 DEBUG [Thread-1647] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:56,605 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:56,605 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:56,605 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:56,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:56,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:56,705 DEBUG [Thread-1647] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:56,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:56,707 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:56,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:56,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:56,708 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:56,709 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:56,709 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:56,709 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:56,709 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:56,709 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:56,710 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:56,710 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:56,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:56,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:56,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:56,711 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:56,711 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:56,711 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:56,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:56,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:56,711 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:56,711 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:56,712 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:56,712 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:56,712 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:56,712 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:56,712 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:56,712 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:56,712 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:56,712 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. started... 2023-04-12 15:54:56,712 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing deda1ce51fef7ef8d26da255318c55e4 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-12 15:54:56,712 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK deda1ce51fef7ef8d26da255318c55e4, store=A 2023-04-12 15:54:56,712 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:56,712 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK deda1ce51fef7ef8d26da255318c55e4, store=B 2023-04-12 15:54:56,712 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:56,713 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK deda1ce51fef7ef8d26da255318c55e4, store=C 2023-04-12 15:54:56,713 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:56,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on deda1ce51fef7ef8d26da255318c55e4 2023-04-12 15:54:56,893 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. as already flushing 2023-04-12 15:54:56,944 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681314896912, max timestamp=1681314896943Num uniques 859; 2023-04-12 15:54:56,944 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681314896912, max timestamp=1681314896943Num uniques 859; 2023-04-12 15:54:56,945 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681314896912, max timestamp=1681314896943Num uniques 859; 2023-04-12 15:54:56,944 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-12 15:54:56,945 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-12 15:54:56,945 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-12 15:54:56,990 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:56,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 297 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53012 deadline: 1681314956988, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:57,093 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:57,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 299 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53012 deadline: 1681314957091, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:57,121 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=493 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/A/be4c85cc009945c180a948cc04ff66e0 2023-04-12 15:54:57,130 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=493 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/B/0b26020e1a0f49a3ba814c3b71cac446 2023-04-12 15:54:57,296 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:57,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 301 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53012 deadline: 1681314957294, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:57,539 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=493 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/C/dbaf57c12f3e4b269df648a537986792 2023-04-12 15:54:57,544 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/A/be4c85cc009945c180a948cc04ff66e0 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/be4c85cc009945c180a948cc04ff66e0 2023-04-12 15:54:57,547 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/be4c85cc009945c180a948cc04ff66e0, entries=100, sequenceid=493, filesize=9.5 K 2023-04-12 15:54:57,548 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/B/0b26020e1a0f49a3ba814c3b71cac446 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/0b26020e1a0f49a3ba814c3b71cac446 2023-04-12 15:54:57,554 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/0b26020e1a0f49a3ba814c3b71cac446, entries=100, sequenceid=493, filesize=9.5 K 2023-04-12 15:54:57,555 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/C/dbaf57c12f3e4b269df648a537986792 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/dbaf57c12f3e4b269df648a537986792 2023-04-12 15:54:57,558 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/dbaf57c12f3e4b269df648a537986792, entries=100, sequenceid=493, filesize=9.5 K 2023-04-12 15:54:57,559 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for deda1ce51fef7ef8d26da255318c55e4 in 847ms, sequenceid=493, compaction requested=true 2023-04-12 15:54:57,559 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for deda1ce51fef7ef8d26da255318c55e4: 2023-04-12 15:54:57,559 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:57,559 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:57,559 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:57,559 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:57,559 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:57,559 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:57,562 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:57,562 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:57,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:57,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:57,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:57,562 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:57,562 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:57,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:57,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:57,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:57,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:57,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:57,564 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:57,564 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@395b27b3[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:57,564 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:57,564 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:57,565 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:57,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:57,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:57,565 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:57,565 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:57,565 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:57,565 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:57,566 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:57,567 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:57,567 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:57,567 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:57,567 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:57,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:57,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:57,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:57,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:57,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:57,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:57,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:57,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:57,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:57,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:57,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:57,570 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:57,570 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:57,570 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:57,570 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:57,570 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:57,570 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:57,571 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:57,571 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:57,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:57,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:57,571 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:57,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:57,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:57,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:57,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:57,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:57,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:57,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:57,572 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:57,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:57,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:57,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:57,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:57,573 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:57,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:57,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:57,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:57,577 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:57,577 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:57,577 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:57,577 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:57,577 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:57,577 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:57,577 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:57,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:57,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:57,577 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:57,577 DEBUG [Thread-1647] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:57,577 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:57,577 DEBUG [Thread-1647] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:57,577 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:57,577 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:57,577 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:57,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:57,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:57,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on deda1ce51fef7ef8d26da255318c55e4 2023-04-12 15:54:57,599 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing deda1ce51fef7ef8d26da255318c55e4 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-12 15:54:57,599 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK deda1ce51fef7ef8d26da255318c55e4, store=A 2023-04-12 15:54:57,599 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:54:57,599 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK deda1ce51fef7ef8d26da255318c55e4, store=B 2023-04-12 15:54:57,599 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:54:57,599 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK deda1ce51fef7ef8d26da255318c55e4, store=C 2023-04-12 15:54:57,599 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:54:57,629 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:57,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 311 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53012 deadline: 1681314957626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:57,677 DEBUG [Thread-1647] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:57,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:57,679 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:57,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:57,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:57,680 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:57,680 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:57,681 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:57,681 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:57,681 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:57,681 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:57,682 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:57,682 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:57,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:57,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:57,683 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:57,683 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:57,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:57,683 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:57,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:57,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:57,683 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:57,684 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:57,684 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:57,684 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:57,684 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:57,684 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:57,684 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:57,684 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:57,684 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:57,684 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. started... 2023-04-12 15:54:57,684 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. as already flushing 2023-04-12 15:54:57,684 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:57,684 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:57,684 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:57,684 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:57,685 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:57,685 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:57,686 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:57,686 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:57,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:57,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:57,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:57,686 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:57,686 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:57,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:57,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:57,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:57,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:57,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:57,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:57,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5cf16060[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:57,688 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:57,688 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:57,689 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:57,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:57,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:57,689 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:57,689 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:57,689 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:57,689 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:57,689 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:57,691 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:57,691 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:57,691 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:57,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:57,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:57,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:57,692 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:57,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:57,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:57,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:57,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:57,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:57,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:57,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:57,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:57,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:57,694 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:57,695 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:57,695 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:57,695 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:57,695 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:57,696 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:57,696 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:57,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:57,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:57,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:57,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:57,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:57,696 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:57,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:57,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:57,697 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:57,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:57,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:57,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:57,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:57,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:57,698 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:57,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:57,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:57,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:57,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:57,699 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:57,699 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:57,699 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:57,700 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:57,699 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:57,700 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:57,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:57,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:57,700 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:57,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:57,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:57,700 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:57,700 DEBUG [Thread-1647] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:57,700 DEBUG [Thread-1647] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:57,700 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:57,700 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:57,700 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:57,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:57,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:57,732 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:57,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 313 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53012 deadline: 1681314957730, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:57,800 DEBUG [Thread-1647] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:57,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:57,802 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:57,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:57,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:57,803 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:57,803 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:57,803 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:57,803 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:57,804 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:57,804 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:57,808 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:57,808 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:57,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:57,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:57,808 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:57,808 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:57,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:57,808 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:57,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:57,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:57,809 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:57,809 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:57,809 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:57,809 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:57,809 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:57,809 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:57,809 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:57,809 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:57,809 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:57,809 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. started... 2023-04-12 15:54:57,810 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. as already flushing 2023-04-12 15:54:57,810 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:57,810 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:57,810 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:57,810 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:57,810 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:57,810 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:57,811 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:57,811 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:57,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:57,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:57,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:57,812 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:57,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:57,812 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:57,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:57,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:57,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:57,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:57,813 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:57,813 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38333823[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:57,813 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:57,813 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:57,814 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:57,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:57,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:57,814 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:57,814 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:57,814 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:57,814 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:57,814 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:57,817 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:57,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:57,817 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:57,817 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:57,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:57,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:57,817 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:57,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:57,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:57,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:57,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:57,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:57,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:57,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:57,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:57,819 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:57,819 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:57,819 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:57,819 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:57,819 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:57,819 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:57,821 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:57,821 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:57,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:57,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:57,821 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:57,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:57,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:57,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:57,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:57,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:57,821 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:57,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:57,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:57,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:57,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:57,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:57,822 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:57,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:57,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:57,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:57,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:57,824 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:57,824 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:57,824 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:57,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:57,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:57,825 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:57,824 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:57,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:57,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:57,824 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:57,825 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:57,825 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:57,825 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:57,825 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:57,825 DEBUG [Thread-1647] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:57,825 DEBUG [Thread-1647] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:57,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:57,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:57,825 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:57,925 DEBUG [Thread-1647] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:57,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:57,927 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:57,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:57,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:57,928 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:57,928 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:57,928 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:57,928 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:57,929 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:57,929 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:57,930 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:57,930 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:57,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:57,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:57,930 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:57,930 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:57,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:57,931 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:57,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:57,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:57,931 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:57,931 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:57,931 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:57,931 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:57,932 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:57,932 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:57,932 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:57,932 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:57,932 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:57,932 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. started... 2023-04-12 15:54:57,932 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. as already flushing 2023-04-12 15:54:57,932 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:57,932 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:57,932 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:57,932 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:57,932 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:57,932 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:57,934 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:57,934 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:57,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:57,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:57,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:57,935 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:57,935 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:57,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:57,935 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:57,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:57,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 315 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53012 deadline: 1681314957934, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:57,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:57,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:57,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:57,936 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:57,937 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@882e77f[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:57,937 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:57,937 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:57,938 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:57,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:57,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:57,938 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:57,938 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:57,938 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:57,938 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:57,938 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:57,939 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:57,939 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:57,939 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:57,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:57,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:57,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:57,940 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:57,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:57,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:57,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:57,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:57,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:57,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:57,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:57,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:57,942 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:57,942 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:57,942 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:57,942 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:57,942 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:57,942 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:57,944 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:57,944 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:57,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:57,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:57,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:57,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:57,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:57,944 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:57,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:57,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:57,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:57,945 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:57,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:57,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:57,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:57,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:57,946 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:57,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:57,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:57,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:57,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:57,948 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:57,948 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:57,949 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:57,949 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:57,949 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:57,949 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:57,949 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:57,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:57,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:57,949 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:57,949 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:57,949 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:57,949 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:57,949 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:57,949 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:57,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:57,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:57,949 DEBUG [Thread-1647] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:57,949 DEBUG [Thread-1647] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:58,017 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=530 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/A/f33ae68e49164c1b84ded121bf43e366 2023-04-12 15:54:58,029 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=530 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/B/6bbb97651c264e0b8fcc9d7a7d513ee0 2023-04-12 15:54:58,043 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=530 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/C/0d21740358114e759f24948a2a314b38 2023-04-12 15:54:58,048 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/A/f33ae68e49164c1b84ded121bf43e366 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/f33ae68e49164c1b84ded121bf43e366 2023-04-12 15:54:58,049 DEBUG [Thread-1647] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:58,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:58,051 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/f33ae68e49164c1b84ded121bf43e366, entries=200, sequenceid=530, filesize=14.3 K 2023-04-12 15:54:58,052 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:58,052 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/B/6bbb97651c264e0b8fcc9d7a7d513ee0 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/6bbb97651c264e0b8fcc9d7a7d513ee0 2023-04-12 15:54:58,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:58,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:58,053 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:58,053 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:58,053 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:58,053 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:58,053 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:58,054 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:58,055 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,055 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:58,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:58,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:58,055 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,055 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:58,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:58,056 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:58,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:58,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:58,056 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:58,056 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:58,056 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:58,056 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/6bbb97651c264e0b8fcc9d7a7d513ee0, entries=150, sequenceid=530, filesize=11.9 K 2023-04-12 15:54:58,056 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:58,056 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:58,056 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:58,057 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:58,057 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:58,057 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:58,057 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. started... 2023-04-12 15:54:58,057 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. as already flushing 2023-04-12 15:54:58,057 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:58,057 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:58,057 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:58,057 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,057 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:58,057 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:58,058 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/C/0d21740358114e759f24948a2a314b38 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/0d21740358114e759f24948a2a314b38 2023-04-12 15:54:58,058 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:58,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:58,058 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:58,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:58,059 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:58,059 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:58,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:58,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:58,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:58,061 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:58,061 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68842f63[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:58,061 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:58,061 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:58,062 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/0d21740358114e759f24948a2a314b38, entries=150, sequenceid=530, filesize=11.9 K 2023-04-12 15:54:58,062 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:58,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:58,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:58,062 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:58,062 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:58,062 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:58,062 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,062 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:58,063 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for deda1ce51fef7ef8d26da255318c55e4 in 464ms, sequenceid=530, compaction requested=true 2023-04-12 15:54:58,063 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for deda1ce51fef7ef8d26da255318c55e4: 2023-04-12 15:54:58,063 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:54:58,063 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-12 15:54:58,063 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-12 15:54:58,063 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:54:58,063 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:54:58,064 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:58,064 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:58,064 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:58,064 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:58,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:58,065 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 56894 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-12 15:54:58,065 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59334 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-12 15:54:58,065 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1912): deda1ce51fef7ef8d26da255318c55e4/B is initiating minor compaction (all files) 2023-04-12 15:54:58,065 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1912): deda1ce51fef7ef8d26da255318c55e4/A is initiating minor compaction (all files) 2023-04-12 15:54:58,065 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2259): Starting compaction of deda1ce51fef7ef8d26da255318c55e4/B in TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:58,065 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of deda1ce51fef7ef8d26da255318c55e4/A in TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:58,065 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/65fb7b97edba4ca88b4b69faf2b89d73, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/6f13f9cc00884607abb5418d7a5c234c, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/d978b91e475f436db70e31596c9b65d9, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/0b26020e1a0f49a3ba814c3b71cac446, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/6bbb97651c264e0b8fcc9d7a7d513ee0] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp, totalSize=55.6 K 2023-04-12 15:54:58,065 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/b76e0fd08e50492eb7b5afe7b85fa74c, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/986287c66bc8405a8f5a2cf776e27c2b, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/69060ab698c841fab0c5f63b3b261569, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/be4c85cc009945c180a948cc04ff66e0, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/f33ae68e49164c1b84ded121bf43e366] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp, totalSize=57.9 K 2023-04-12 15:54:58,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:58,065 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 65fb7b97edba4ca88b4b69faf2b89d73, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=432, earliestPutTs=1681314893586 2023-04-12 15:54:58,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:58,066 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting b76e0fd08e50492eb7b5afe7b85fa74c, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=432, earliestPutTs=1681314893586 2023-04-12 15:54:58,066 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 6f13f9cc00884607abb5418d7a5c234c, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=444, earliestPutTs=1681314894232 2023-04-12 15:54:58,066 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 986287c66bc8405a8f5a2cf776e27c2b, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=444, earliestPutTs=1681314894232 2023-04-12 15:54:58,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:58,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:58,067 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting d978b91e475f436db70e31596c9b65d9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=482, earliestPutTs=1681314895424 2023-04-12 15:54:58,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:58,067 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 69060ab698c841fab0c5f63b3b261569, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=482, earliestPutTs=1681314895424 2023-04-12 15:54:58,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,067 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 0b26020e1a0f49a3ba814c3b71cac446, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=493, earliestPutTs=1681314895765 2023-04-12 15:54:58,068 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting be4c85cc009945c180a948cc04ff66e0, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=493, earliestPutTs=1681314895765 2023-04-12 15:54:58,068 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 6bbb97651c264e0b8fcc9d7a7d513ee0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=530, earliestPutTs=1681314896978 2023-04-12 15:54:58,068 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting f33ae68e49164c1b84ded121bf43e366, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=530, earliestPutTs=1681314896978 2023-04-12 15:54:58,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:58,068 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:58,068 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:58,068 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:58,068 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:58,068 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:58,069 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:58,070 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:58,070 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:58,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:58,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:58,070 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:58,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:58,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:58,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:58,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:58,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:58,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:58,071 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:58,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:58,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:58,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:58,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,073 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:58,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:58,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,076 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:58,076 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:58,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:58,076 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:58,076 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:58,076 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:58,076 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,076 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:58,076 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:58,076 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:58,076 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:58,076 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:58,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:58,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:58,076 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:58,077 DEBUG [Thread-1647] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:58,077 DEBUG [Thread-1647] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:58,084 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] throttle.PressureAwareThroughputController(145): deda1ce51fef7ef8d26da255318c55e4#A#compaction#292 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:54:58,084 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] throttle.PressureAwareThroughputController(145): deda1ce51fef7ef8d26da255318c55e4#B#compaction#291 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:54:58,177 DEBUG [Thread-1647] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:58,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:58,179 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:58,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:58,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:58,180 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:58,180 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:58,180 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:58,180 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:58,180 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:58,180 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:58,182 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:58,182 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:58,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:58,182 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,182 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:58,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:58,182 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:58,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:58,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:58,183 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:58,183 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:58,183 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:58,183 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:58,183 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:58,183 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:58,183 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:58,183 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:58,183 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:58,183 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. started... 2023-04-12 15:54:58,183 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing deda1ce51fef7ef8d26da255318c55e4 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-12 15:54:58,184 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK deda1ce51fef7ef8d26da255318c55e4, store=A 2023-04-12 15:54:58,184 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:58,184 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK deda1ce51fef7ef8d26da255318c55e4, store=B 2023-04-12 15:54:58,184 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:58,184 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK deda1ce51fef7ef8d26da255318c55e4, store=C 2023-04-12 15:54:58,184 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:58,191 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=540 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/A/cabda8b29e3f43ef9132081cea19fefd 2023-04-12 15:54:58,201 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=540 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/B/bcea047f6e7b43c99aa9a532533a45a6 2023-04-12 15:54:58,211 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=540 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/C/1c14fa96704c45b5abec6068164ef66c 2023-04-12 15:54:58,216 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/A/cabda8b29e3f43ef9132081cea19fefd as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/cabda8b29e3f43ef9132081cea19fefd 2023-04-12 15:54:58,219 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/cabda8b29e3f43ef9132081cea19fefd, entries=150, sequenceid=540, filesize=11.9 K 2023-04-12 15:54:58,220 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/B/bcea047f6e7b43c99aa9a532533a45a6 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/bcea047f6e7b43c99aa9a532533a45a6 2023-04-12 15:54:58,224 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/bcea047f6e7b43c99aa9a532533a45a6, entries=150, sequenceid=540, filesize=11.9 K 2023-04-12 15:54:58,225 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/C/1c14fa96704c45b5abec6068164ef66c as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/1c14fa96704c45b5abec6068164ef66c 2023-04-12 15:54:58,229 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/1c14fa96704c45b5abec6068164ef66c, entries=150, sequenceid=540, filesize=11.9 K 2023-04-12 15:54:58,230 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=0 B/0 for deda1ce51fef7ef8d26da255318c55e4 in 47ms, sequenceid=540, compaction requested=true 2023-04-12 15:54:58,230 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for deda1ce51fef7ef8d26da255318c55e4: 2023-04-12 15:54:58,230 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:58,230 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:58,230 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:58,230 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,235 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:58,235 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:58,240 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,240 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:58,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:58,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:58,240 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:58,240 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:58,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:58,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:58,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:58,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:58,241 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:58,241 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3fa1a073[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:58,241 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:58,241 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:58,242 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:58,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:58,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:58,243 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:58,243 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:58,243 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:58,243 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,243 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:58,244 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,244 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:58,244 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:58,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:58,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:58,244 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:58,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:58,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:58,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:58,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:58,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:58,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:58,247 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:58,247 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:58,247 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:58,247 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:58,247 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:58,247 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:58,249 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:58,249 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:58,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:58,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:58,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:58,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:58,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:58,249 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:58,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:58,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:58,250 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:58,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:58,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:58,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:58,252 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:58,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on deda1ce51fef7ef8d26da255318c55e4 2023-04-12 15:54:58,254 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing deda1ce51fef7ef8d26da255318c55e4 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-12 15:54:58,254 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK deda1ce51fef7ef8d26da255318c55e4, store=A 2023-04-12 15:54:58,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,254 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:58,254 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK deda1ce51fef7ef8d26da255318c55e4, store=B 2023-04-12 15:54:58,254 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:58,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:58,254 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK deda1ce51fef7ef8d26da255318c55e4, store=C 2023-04-12 15:54:58,255 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:58,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:58,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,258 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:58,258 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:58,258 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:58,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:58,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:58,258 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:58,258 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:58,258 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:58,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:58,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:58,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:58,258 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:58,258 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,258 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:58,259 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:58,259 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:58,258 DEBUG [Thread-1647] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:58,259 DEBUG [Thread-1647] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:58,316 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681314898286, max timestamp=1681314898315Num uniques 859; 2023-04-12 15:54:58,317 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-12 15:54:58,317 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681314898286, max timestamp=1681314898315Num uniques 859; 2023-04-12 15:54:58,317 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-12 15:54:58,317 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681314898286, max timestamp=1681314898315Num uniques 859; 2023-04-12 15:54:58,317 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-12 15:54:58,359 DEBUG [Thread-1647] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:58,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:58,361 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:58,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:58,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:58,362 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:58,362 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:58,362 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:58,362 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:58,363 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:58,363 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:58,364 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:58,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:58,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:58,364 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:58,365 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,365 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:58,365 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:58,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:58,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:58,365 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:58,365 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:58,365 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:58,365 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:58,365 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:58,365 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:58,366 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:58,366 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:58,366 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:58,366 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. started... 2023-04-12 15:54:58,366 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. as already flushing 2023-04-12 15:54:58,366 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:58,366 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:58,366 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:58,366 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,366 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:58,366 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:58,367 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:58,367 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:58,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:58,368 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:58,368 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:58,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:58,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:58,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:58,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:58,369 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:58,369 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ef27584[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:58,369 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:58,369 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:58,370 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:58,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:58,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:58,371 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:58,371 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:58,371 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:58,371 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,371 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:58,373 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,373 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:58,373 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:58,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:58,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:58,373 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:58,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:58,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:58,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:58,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:58,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:58,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:58,375 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:58,375 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:58,375 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:58,375 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:58,375 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:58,375 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:58,377 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:58,377 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:58,377 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:58,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:58,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:58,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:58,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:58,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:58,377 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:58,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:58,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:58,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:58,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:58,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:58,378 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:58,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:58,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:58,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,380 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,380 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:58,380 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:58,380 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:58,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:58,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:58,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:58,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:58,380 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:58,380 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:58,380 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,381 DEBUG [Thread-1647] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:58,380 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:58,381 DEBUG [Thread-1647] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:58,381 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:58,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:58,381 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:58,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:58,381 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:58,396 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:58,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 358 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53012 deadline: 1681314958394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,481 DEBUG [Thread-1647] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:58,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:58,483 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:58,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:58,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:58,484 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:58,484 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:58,484 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:58,484 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:58,484 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:58,484 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:58,485 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,485 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:58,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:58,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:58,486 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,486 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:58,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:58,486 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:58,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:58,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:58,487 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:58,487 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:58,487 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:58,487 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:58,487 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:58,487 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:58,487 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:58,487 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:58,487 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:58,487 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. started... 2023-04-12 15:54:58,487 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. as already flushing 2023-04-12 15:54:58,487 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:58,488 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:58,488 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:58,488 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,488 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:58,488 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:58,492 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:58,492 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:58,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:58,492 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:58,492 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:58,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:58,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:58,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:58,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:58,494 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:58,494 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d4c03a[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:58,494 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:58,494 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:58,497 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:58,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:58,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:58,497 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:58,497 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:58,497 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,497 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:58,497 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:58,499 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:58,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 360 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53012 deadline: 1681314958497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,500 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:58,500 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:58,500 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:58,500 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,500 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/B/12d580cd11eb4fca8eb8f1b48d2c6e4c as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/12d580cd11eb4fca8eb8f1b48d2c6e4c 2023-04-12 15:54:58,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:58,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:58,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:58,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:58,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:58,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:58,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:58,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:58,517 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:58,517 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:58,517 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:58,517 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:58,517 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:58,517 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:58,519 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/A/bc97ef723aa34d9fb828dd5b598b6836 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/bc97ef723aa34d9fb828dd5b598b6836 2023-04-12 15:54:58,519 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:58,519 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:58,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:58,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:58,519 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:58,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:58,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:58,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:58,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:58,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:58,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:58,520 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:58,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:58,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:58,521 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in deda1ce51fef7ef8d26da255318c55e4/B of deda1ce51fef7ef8d26da255318c55e4 into 12d580cd11eb4fca8eb8f1b48d2c6e4c(size=12.9 K), total size for store is 24.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:54:58,522 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for deda1ce51fef7ef8d26da255318c55e4: 2023-04-12 15:54:58,522 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4., storeName=deda1ce51fef7ef8d26da255318c55e4/B, priority=11, startTime=1681314898063; duration=0sec 2023-04-12 15:54:58,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:58,522 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:54:58,522 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-12 15:54:58,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,522 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:58,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:58,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,525 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 69080 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-12 15:54:58,525 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1912): deda1ce51fef7ef8d26da255318c55e4/C is initiating minor compaction (all files) 2023-04-12 15:54:58,525 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2259): Starting compaction of deda1ce51fef7ef8d26da255318c55e4/C in TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:58,525 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/fd9beb5e5eea428e847978b6ba355347, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/45afff1641b64118a1aefd2efb4c0a20, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/1d647b39904642e683f14de2ef3b913b, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/dbaf57c12f3e4b269df648a537986792, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/0d21740358114e759f24948a2a314b38, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/1c14fa96704c45b5abec6068164ef66c] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp, totalSize=67.5 K 2023-04-12 15:54:58,525 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting fd9beb5e5eea428e847978b6ba355347, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=432, earliestPutTs=1681314893586 2023-04-12 15:54:58,526 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,526 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:58,526 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:58,526 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:58,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:58,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:58,526 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,526 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:58,526 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:58,526 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:58,526 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:58,526 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:58,526 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 45afff1641b64118a1aefd2efb4c0a20, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=444, earliestPutTs=1681314894232 2023-04-12 15:54:58,527 DEBUG [Thread-1647] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:58,527 DEBUG [Thread-1647] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:58,527 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:58,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:58,527 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:58,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:58,527 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 1d647b39904642e683f14de2ef3b913b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=482, earliestPutTs=1681314895424 2023-04-12 15:54:58,527 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting dbaf57c12f3e4b269df648a537986792, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=493, earliestPutTs=1681314895765 2023-04-12 15:54:58,528 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 0d21740358114e759f24948a2a314b38, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=530, earliestPutTs=1681314896978 2023-04-12 15:54:58,528 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 1c14fa96704c45b5abec6068164ef66c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=540, earliestPutTs=1681314897619 2023-04-12 15:54:58,532 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in deda1ce51fef7ef8d26da255318c55e4/A of deda1ce51fef7ef8d26da255318c55e4 into bc97ef723aa34d9fb828dd5b598b6836(size=12.9 K), total size for store is 24.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:54:58,532 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for deda1ce51fef7ef8d26da255318c55e4: 2023-04-12 15:54:58,532 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4., storeName=deda1ce51fef7ef8d26da255318c55e4/A, priority=11, startTime=1681314898063; duration=0sec 2023-04-12 15:54:58,532 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:54:58,541 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] throttle.PressureAwareThroughputController(145): deda1ce51fef7ef8d26da255318c55e4#C#compaction#297 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:54:58,627 DEBUG [Thread-1647] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:58,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:58,629 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:58,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:58,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:58,630 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:58,630 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:58,630 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:58,630 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:58,631 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:58,631 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:58,632 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,632 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:58,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:58,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:58,632 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,632 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:58,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:58,633 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:58,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:58,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:58,633 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:58,633 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:58,633 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:58,633 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:58,633 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:58,633 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:58,633 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:58,634 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:58,634 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:58,634 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. started... 2023-04-12 15:54:58,634 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. as already flushing 2023-04-12 15:54:58,634 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:58,634 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:58,634 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:58,634 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,634 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:58,634 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:58,635 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,635 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:58,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:58,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:58,636 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:58,636 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:58,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:58,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:58,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:58,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:58,637 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:58,637 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48e20854[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:58,637 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:58,637 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:58,638 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:58,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:58,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:58,638 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:58,638 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:58,638 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:58,638 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,638 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:58,648 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,648 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:58,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:58,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:58,648 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:58,648 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:58,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:58,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:58,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:58,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:58,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:58,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:58,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:58,650 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:58,651 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:58,651 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:58,651 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:58,651 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:58,652 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:58,652 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:58,652 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:58,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:58,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:58,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:58,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:58,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:58,652 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:58,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:58,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:58,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:58,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:58,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:58,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:58,654 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:58,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:58,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,656 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,656 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:58,656 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:58,656 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:58,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:58,656 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:58,656 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,657 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:58,657 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:58,657 DEBUG [Thread-1647] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:58,657 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:58,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:58,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:58,657 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:58,657 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:58,657 DEBUG [Thread-1647] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:58,657 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:58,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:58,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:58,680 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=550 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/A/d5e62724e7d04940b0a163ceb0398854 2023-04-12 15:54:58,702 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:58,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 362 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53012 deadline: 1681314958700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,757 DEBUG [Thread-1647] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:58,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:58,759 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:58,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:58,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:58,761 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:58,761 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:58,761 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:58,761 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:58,761 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:58,761 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:58,762 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:58,762 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:58,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:58,762 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,763 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:58,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:58,763 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:58,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:58,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:58,763 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:58,763 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:58,763 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:58,763 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:58,764 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:58,764 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:58,765 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:58,765 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:58,765 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:58,765 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. started... 2023-04-12 15:54:58,765 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. as already flushing 2023-04-12 15:54:58,765 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:58,765 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:58,765 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:58,765 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,765 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:58,765 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:58,768 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,768 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:58,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:58,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:58,768 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:58,768 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:58,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:58,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:58,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:58,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:58,770 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:58,770 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a728b03[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:58,770 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:58,770 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:58,771 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:58,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:58,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:58,771 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:58,771 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,771 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:58,771 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:58,771 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:58,773 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:58,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:58,773 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:58,773 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:58,773 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:58,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:58,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:58,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:58,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:58,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:58,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:58,775 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:58,775 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:58,776 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:58,776 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:58,776 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:58,776 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:58,777 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:58,777 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:58,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:58,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:58,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:58,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:58,777 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:58,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:58,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:58,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:58,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:58,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:58,778 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:58,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:58,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:58,780 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:58,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:58,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,786 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,786 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:58,786 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:58,786 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:58,786 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:58,786 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:58,786 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:58,786 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:58,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:58,786 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:58,786 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:58,787 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:58,787 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:58,787 DEBUG [Thread-1647] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:58,787 DEBUG [Thread-1647] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:58,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:58,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:58,789 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:58,870 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:58,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53014 deadline: 1681314958869, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,871 DEBUG [Thread-1637] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8173 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4., hostname=jenkins-hbase4.apache.org,38905,1681314818739, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-12 15:54:58,878 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:58,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53032 deadline: 1681314958877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,880 DEBUG [Thread-1641] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8181 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4., hostname=jenkins-hbase4.apache.org,38905,1681314818739, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-12 15:54:58,887 DEBUG [Thread-1647] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:58,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:58,889 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:58,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:58,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:58,890 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:58,891 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:58,891 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:58,891 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:58,891 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:58,891 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:58,892 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:58,892 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:58,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:58,892 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,893 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:58,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:58,893 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:58,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:58,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:58,893 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:58,893 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:58,893 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:58,894 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:58,894 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:58,894 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:58,894 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:58,894 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:58,894 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:58,894 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. started... 2023-04-12 15:54:58,894 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. as already flushing 2023-04-12 15:54:58,894 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:58,894 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:58,894 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:58,894 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,894 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:58,894 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:58,895 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:58,896 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:58,896 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:58,896 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:58,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:58,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:58,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:58,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:58,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:58,898 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:58,898 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77219f9c[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:58,898 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:58,898 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:58,899 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:58,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:58,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:58,899 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:58,899 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:58,899 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:58,899 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,899 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:58,901 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:58,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:58,901 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:58,902 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:58,902 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:58,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:58,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:58,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:58,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:58,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:58,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:58,904 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:58,904 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:58,904 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:58,904 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:58,904 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:58,904 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:58,906 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:58,906 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:58,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:58,906 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:58,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:58,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:58,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:58,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:58,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:58,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:58,906 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:58,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:58,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:58,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:58,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:58,907 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:58,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:58,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,910 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,910 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:58,910 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:58,910 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:58,910 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:58,910 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:58,910 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:58,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:58,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:58,910 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:58,910 DEBUG [Thread-1647] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:58,910 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:58,910 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:58,910 DEBUG [Thread-1647] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:58,910 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:58,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:58,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:58,910 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:58,911 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:58,958 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/C/163142a310894c1fa1bb9762d24ace76 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/163142a310894c1fa1bb9762d24ace76 2023-04-12 15:54:58,962 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in deda1ce51fef7ef8d26da255318c55e4/C of deda1ce51fef7ef8d26da255318c55e4 into 163142a310894c1fa1bb9762d24ace76(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:54:58,962 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for deda1ce51fef7ef8d26da255318c55e4: 2023-04-12 15:54:58,962 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4., storeName=deda1ce51fef7ef8d26da255318c55e4/C, priority=10, startTime=1681314898063; duration=0sec 2023-04-12 15:54:58,962 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:54:59,005 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:59,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 364 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53012 deadline: 1681314959004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:59,010 DEBUG [Thread-1647] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:59,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:59,012 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:59,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:59,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:59,014 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:59,014 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:59,014 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:59,014 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:59,014 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:59,014 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:59,015 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:59,015 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:59,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:59,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:59,016 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:59,016 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:59,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:59,016 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:59,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:59,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:59,016 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:59,017 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:59,017 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:59,017 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:59,017 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:59,017 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:59,017 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:59,017 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:59,017 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:59,017 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. started... 2023-04-12 15:54:59,017 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. as already flushing 2023-04-12 15:54:59,017 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:59,017 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:59,017 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:59,017 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:59,017 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:59,017 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:59,020 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:59,020 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:59,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:59,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:59,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:59,020 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:59,020 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:59,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:59,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:59,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:59,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:59,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:59,022 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:59,022 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50534452[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:59,022 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:59,022 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:59,023 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:59,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:59,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:59,023 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:59,023 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:59,023 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:59,023 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:59,023 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:59,024 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:59,024 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:59,024 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:59,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:59,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:59,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:59,024 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:59,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:59,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:59,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:59,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:59,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:59,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:59,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:59,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:59,027 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:59,027 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:59,027 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:59,027 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:59,027 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:59,027 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:59,032 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:59,032 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:59,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:59,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:59,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:59,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:59,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:59,032 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:59,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:59,032 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:59,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:59,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:59,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:59,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:59,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:59,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:59,034 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:59,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:59,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:59,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:59,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:59,036 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:59,036 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:59,036 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:59,036 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:59,036 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:59,036 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:59,036 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:59,036 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:59,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:59,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:59,036 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:59,036 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:59,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:59,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:59,036 DEBUG [Thread-1647] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:59,036 DEBUG [Thread-1647] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:59,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:59,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:59,090 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=550 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/B/215b3e22f23a4074a2167e2e3c192bf6 2023-04-12 15:54:59,099 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=550 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/C/24bdbaef4f404fd4aa27d4ab52ac22ff 2023-04-12 15:54:59,113 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/A/d5e62724e7d04940b0a163ceb0398854 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/d5e62724e7d04940b0a163ceb0398854 2023-04-12 15:54:59,116 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/d5e62724e7d04940b0a163ceb0398854, entries=200, sequenceid=550, filesize=14.3 K 2023-04-12 15:54:59,117 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/B/215b3e22f23a4074a2167e2e3c192bf6 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/215b3e22f23a4074a2167e2e3c192bf6 2023-04-12 15:54:59,120 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/215b3e22f23a4074a2167e2e3c192bf6, entries=150, sequenceid=550, filesize=11.9 K 2023-04-12 15:54:59,121 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/C/24bdbaef4f404fd4aa27d4ab52ac22ff as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/24bdbaef4f404fd4aa27d4ab52ac22ff 2023-04-12 15:54:59,132 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/24bdbaef4f404fd4aa27d4ab52ac22ff, entries=150, sequenceid=550, filesize=11.9 K 2023-04-12 15:54:59,133 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for deda1ce51fef7ef8d26da255318c55e4 in 879ms, sequenceid=550, compaction requested=true 2023-04-12 15:54:59,133 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for deda1ce51fef7ef8d26da255318c55e4: 2023-04-12 15:54:59,133 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:54:59,133 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:54:59,133 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 15:54:59,133 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:54:59,133 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 15:54:59,134 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40020 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 15:54:59,134 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37580 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 15:54:59,134 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1912): deda1ce51fef7ef8d26da255318c55e4/A is initiating minor compaction (all files) 2023-04-12 15:54:59,134 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1912): deda1ce51fef7ef8d26da255318c55e4/B is initiating minor compaction (all files) 2023-04-12 15:54:59,134 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of deda1ce51fef7ef8d26da255318c55e4/A in TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:59,134 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2259): Starting compaction of deda1ce51fef7ef8d26da255318c55e4/B in TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:59,134 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/bc97ef723aa34d9fb828dd5b598b6836, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/cabda8b29e3f43ef9132081cea19fefd, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/d5e62724e7d04940b0a163ceb0398854] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp, totalSize=39.1 K 2023-04-12 15:54:59,135 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/12d580cd11eb4fca8eb8f1b48d2c6e4c, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/bcea047f6e7b43c99aa9a532533a45a6, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/215b3e22f23a4074a2167e2e3c192bf6] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp, totalSize=36.7 K 2023-04-12 15:54:59,135 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting bc97ef723aa34d9fb828dd5b598b6836, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=530, earliestPutTs=1681314896978 2023-04-12 15:54:59,135 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 12d580cd11eb4fca8eb8f1b48d2c6e4c, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=530, earliestPutTs=1681314896978 2023-04-12 15:54:59,135 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting cabda8b29e3f43ef9132081cea19fefd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=540, earliestPutTs=1681314897619 2023-04-12 15:54:59,135 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting bcea047f6e7b43c99aa9a532533a45a6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=540, earliestPutTs=1681314897619 2023-04-12 15:54:59,135 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting d5e62724e7d04940b0a163ceb0398854, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=550, earliestPutTs=1681314898244 2023-04-12 15:54:59,136 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 215b3e22f23a4074a2167e2e3c192bf6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=550, earliestPutTs=1681314898244 2023-04-12 15:54:59,136 DEBUG [Thread-1647] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:59,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:59,139 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:59,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:59,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:59,152 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:59,152 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:59,152 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:59,152 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:59,153 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:59,153 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:59,154 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] throttle.PressureAwareThroughputController(145): deda1ce51fef7ef8d26da255318c55e4#A#compaction#300 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:54:59,154 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] throttle.PressureAwareThroughputController(145): deda1ce51fef7ef8d26da255318c55e4#B#compaction#301 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:54:59,154 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:59,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:59,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:59,154 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:59,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:59,155 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:59,155 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:59,155 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:59,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:59,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:59,155 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:59,156 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:59,156 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:59,156 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:59,156 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:59,156 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:59,156 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:59,156 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:59,156 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. started... 2023-04-12 15:54:59,156 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing deda1ce51fef7ef8d26da255318c55e4 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-12 15:54:59,156 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK deda1ce51fef7ef8d26da255318c55e4, store=A 2023-04-12 15:54:59,156 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:54:59,157 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK deda1ce51fef7ef8d26da255318c55e4, store=B 2023-04-12 15:54:59,157 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:54:59,157 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK deda1ce51fef7ef8d26da255318c55e4, store=C 2023-04-12 15:54:59,157 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:54:59,183 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/A/235f1d9d21b449cea481d1c5cccb2dd0 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/235f1d9d21b449cea481d1c5cccb2dd0 2023-04-12 15:54:59,188 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in deda1ce51fef7ef8d26da255318c55e4/A of deda1ce51fef7ef8d26da255318c55e4 into 235f1d9d21b449cea481d1c5cccb2dd0(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:54:59,188 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for deda1ce51fef7ef8d26da255318c55e4: 2023-04-12 15:54:59,188 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4., storeName=deda1ce51fef7ef8d26da255318c55e4/A, priority=13, startTime=1681314899133; duration=0sec 2023-04-12 15:54:59,188 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:54:59,188 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-12 15:54:59,189 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=590 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/A/f58940cf4d934ab592fe1ad52b14b95b 2023-04-12 15:54:59,189 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-12 15:54:59,189 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-12 15:54:59,189 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. because compaction request was cancelled 2023-04-12 15:54:59,190 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/B/fb1a965435e64a5e9fc778f407e579fc as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/fb1a965435e64a5e9fc778f407e579fc 2023-04-12 15:54:59,195 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in deda1ce51fef7ef8d26da255318c55e4/B of deda1ce51fef7ef8d26da255318c55e4 into fb1a965435e64a5e9fc778f407e579fc(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:54:59,195 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for deda1ce51fef7ef8d26da255318c55e4: 2023-04-12 15:54:59,196 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4., storeName=deda1ce51fef7ef8d26da255318c55e4/B, priority=13, startTime=1681314899133; duration=0sec 2023-04-12 15:54:59,196 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:54:59,199 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=590 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/B/91bdd2217398440890aa16cc4d15f358 2023-04-12 15:54:59,208 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=590 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/C/204c40eee97149059460ab8b59b7de12 2023-04-12 15:54:59,212 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/A/f58940cf4d934ab592fe1ad52b14b95b as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/f58940cf4d934ab592fe1ad52b14b95b 2023-04-12 15:54:59,215 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/f58940cf4d934ab592fe1ad52b14b95b, entries=150, sequenceid=590, filesize=11.9 K 2023-04-12 15:54:59,216 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/B/91bdd2217398440890aa16cc4d15f358 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/91bdd2217398440890aa16cc4d15f358 2023-04-12 15:54:59,219 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/91bdd2217398440890aa16cc4d15f358, entries=150, sequenceid=590, filesize=11.9 K 2023-04-12 15:54:59,220 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/C/204c40eee97149059460ab8b59b7de12 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/204c40eee97149059460ab8b59b7de12 2023-04-12 15:54:59,225 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/204c40eee97149059460ab8b59b7de12, entries=150, sequenceid=590, filesize=11.9 K 2023-04-12 15:54:59,226 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=0 B/0 for deda1ce51fef7ef8d26da255318c55e4 in 69ms, sequenceid=590, compaction requested=true 2023-04-12 15:54:59,226 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for deda1ce51fef7ef8d26da255318c55e4: 2023-04-12 15:54:59,226 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:59,226 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:59,226 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:59,226 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:59,227 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:59,227 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:59,228 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:59,228 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:59,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:59,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:59,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:59,228 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:59,228 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:59,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:59,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:59,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:59,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:59,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:59,230 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:59,230 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@82b46a4[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:59,230 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:59,230 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:59,232 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:59,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:59,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:59,232 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:59,232 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:59,232 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:59,232 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:59,232 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:59,233 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:59,233 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:59,233 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:59,233 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:59,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:59,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:59,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:59,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:59,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:59,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:59,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:59,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:59,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:59,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:59,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:59,237 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:59,237 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:59,237 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:59,237 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:59,237 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:59,237 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:59,240 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:59,240 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:59,240 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:59,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:59,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:59,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:59,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:59,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:59,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:59,241 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:59,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:59,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:59,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:59,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:59,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:59,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:59,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:59,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:59,242 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:59,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:59,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:59,244 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:59,244 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:59,244 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:59,244 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:59,244 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:59,245 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:59,245 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:59,245 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:59,244 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:59,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:59,244 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:59,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:59,245 DEBUG [Thread-1647] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:59,245 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:59,245 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:59,245 DEBUG [Thread-1647] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:59,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:59,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:59,345 DEBUG [Thread-1647] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:59,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:59,347 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:59,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:59,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:59,348 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:59,348 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:59,349 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:59,349 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:59,349 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:59,349 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:59,351 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:59,351 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:59,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:59,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:59,351 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:59,351 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:59,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:59,351 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:59,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:59,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:59,352 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:59,352 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:59,352 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:59,352 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:59,352 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:59,352 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:59,352 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:59,352 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:59,352 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:59,352 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. started... 2023-04-12 15:54:59,353 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for deda1ce51fef7ef8d26da255318c55e4: 2023-04-12 15:54:59,353 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:59,353 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:59,353 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:59,353 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:59,353 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:59,353 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:59,354 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:59,354 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:59,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:59,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:59,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:59,355 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:59,355 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:59,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:59,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:59,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:59,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:59,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:59,356 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:59,356 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@666fee31[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:59,356 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:59,356 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:59,358 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:59,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:59,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:59,358 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:59,358 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:59,358 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:59,358 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:59,358 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:59,360 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:59,360 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:59,360 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:59,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:59,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:59,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:59,360 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:59,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:59,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:59,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:59,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:59,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:59,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:59,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:59,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:59,363 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:59,363 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:59,363 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:59,363 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:59,363 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:59,363 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:59,365 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:59,365 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:59,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:59,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:59,365 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:59,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:59,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:59,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:59,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:59,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:59,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:59,365 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:59,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:59,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:59,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:59,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:59,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:59,367 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:59,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:59,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:59,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:59,369 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:59,369 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:59,369 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:59,369 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:59,369 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:59,369 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:59,369 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:59,369 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:59,369 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:59,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:59,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:59,369 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:59,369 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:59,369 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:59,369 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:59,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:59,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:59,370 DEBUG [Thread-1647] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:59,370 DEBUG [Thread-1647] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:59,470 DEBUG [Thread-1647] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:59,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:59,472 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:59,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:59,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:59,473 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:59,473 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:59,474 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:59,474 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:59,474 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:59,474 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:59,476 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:59,476 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:59,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:59,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:59,477 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:59,477 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:59,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:59,477 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:59,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:59,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:59,478 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:59,478 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:59,478 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:59,478 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:59,478 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:59,478 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:59,478 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:59,478 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:59,479 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:59,479 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. started... 2023-04-12 15:54:59,479 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for deda1ce51fef7ef8d26da255318c55e4: 2023-04-12 15:54:59,479 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:59,479 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:59,479 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:59,479 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:59,479 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:59,479 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:59,480 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:59,480 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:59,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:59,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:59,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:59,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:59,481 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:59,481 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:59,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:59,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:59,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:59,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:59,482 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:59,482 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d49501f[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:59,482 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:59,482 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:59,483 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:59,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:59,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:59,483 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:59,484 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:59,484 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:59,484 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:59,484 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:59,485 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:59,485 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:59,485 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:59,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:59,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:59,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:59,485 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:59,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:59,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:59,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:59,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:59,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:59,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:59,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:59,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:59,488 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:59,488 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:59,488 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:59,488 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:59,488 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:59,488 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:59,490 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:59,490 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:59,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:59,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:59,490 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:59,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:59,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:59,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:59,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:59,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:59,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:59,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:59,491 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:59,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:59,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:59,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:59,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:59,492 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:59,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:59,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:59,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:59,497 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:59,497 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:59,497 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:59,497 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:59,497 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:59,497 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:59,497 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:59,497 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:59,497 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:59,497 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:59,497 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:59,497 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:59,497 DEBUG [Thread-1647] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:59,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:59,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:59,497 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:59,497 DEBUG [Thread-1647] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:59,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:59,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:59,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on deda1ce51fef7ef8d26da255318c55e4 2023-04-12 15:54:59,519 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing deda1ce51fef7ef8d26da255318c55e4 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-12 15:54:59,520 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK deda1ce51fef7ef8d26da255318c55e4, store=A 2023-04-12 15:54:59,520 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:59,520 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK deda1ce51fef7ef8d26da255318c55e4, store=B 2023-04-12 15:54:59,520 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:59,520 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK deda1ce51fef7ef8d26da255318c55e4, store=C 2023-04-12 15:54:59,520 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:54:59,527 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=604 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/A/fd3be8fb70fa49a09669feaaa0a0b917 2023-04-12 15:54:59,565 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681314899546, max timestamp=1681314899563Num uniques 859; 2023-04-12 15:54:59,565 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-12 15:54:59,565 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681314899546, max timestamp=1681314899563Num uniques 859; 2023-04-12 15:54:59,565 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-12 15:54:59,567 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681314899546, max timestamp=1681314899563Num uniques 859; 2023-04-12 15:54:59,567 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-12 15:54:59,598 DEBUG [Thread-1647] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:59,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:59,600 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:54:59,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:54:59,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:54:59,601 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:54:59,601 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:59,601 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:54:59,601 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:54:59,601 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:59,601 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:59,602 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:59,602 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:59,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:59,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:59,603 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:59,603 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:54:59,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:59,603 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:59,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:54:59,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:59,603 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:54:59,604 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:54:59,604 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:54:59,604 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:54:59,604 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:54:59,604 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:54:59,604 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:54:59,604 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:54:59,604 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:59,604 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. started... 2023-04-12 15:54:59,604 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. as already flushing 2023-04-12 15:54:59,604 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:54:59,604 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:54:59,604 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:54:59,604 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:59,604 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:54:59,604 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:54:59,607 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:59,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:59,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:59,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:59,607 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:59,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:59,607 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:59,607 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:54:59,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:59,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:59,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:59,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:59,608 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:54:59,608 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57f2f630[Count = 0] remaining members to acquire global barrier 2023-04-12 15:54:59,608 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:54:59,608 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:59,610 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:59,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:59,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:59,610 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:54:59,610 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:54:59,610 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:59,610 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:54:59,610 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:54:59,611 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:54:59,611 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:59,611 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:59,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:59,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:59,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:59,611 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:54:59,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:59,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:59,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:59,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:59,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:59,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:59,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:59,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:54:59,614 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:54:59,614 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:54:59,614 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:54:59,614 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:54:59,614 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:54:59,614 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:54:59,615 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:59,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 407 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53012 deadline: 1681314959613, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:59,615 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:59,615 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:59,615 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:59,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:59,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:59,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:54:59,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:54:59,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:59,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:59,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:59,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:54:59,616 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:59,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:59,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:59,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:54:59,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:59,617 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:59,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:59,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:54:59,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:54:59,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:59,621 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:59,621 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:54:59,621 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:59,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:54:59,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:54:59,621 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:54:59,621 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:54:59,621 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:54:59,621 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:54:59,621 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:54:59,621 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:59,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:54:59,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:54:59,621 DEBUG [Thread-1647] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:54:59,621 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:59,621 DEBUG [Thread-1647] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:54:59,621 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:54:59,621 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:54:59,621 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:54:59,680 DEBUG [Thread-1656] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6a98265c to 127.0.0.1:60154 2023-04-12 15:54:59,680 DEBUG [Thread-1656] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 15:54:59,680 DEBUG [Thread-1648] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x433f251d to 127.0.0.1:60154 2023-04-12 15:54:59,681 DEBUG [Thread-1648] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 15:54:59,681 DEBUG [Thread-1654] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5d2b992d to 127.0.0.1:60154 2023-04-12 15:54:59,681 DEBUG [Thread-1654] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 15:54:59,683 DEBUG [Thread-1652] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3b4658f8 to 127.0.0.1:60154 2023-04-12 15:54:59,683 DEBUG [Thread-1652] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 15:54:59,683 DEBUG [Thread-1650] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x66ee0bd0 to 127.0.0.1:60154 2023-04-12 15:54:59,683 DEBUG [Thread-1650] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 15:54:59,716 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:59,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 409 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53012 deadline: 1681314959716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:59,721 DEBUG [Thread-1647] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:54:59,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:54:59,918 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:59,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 411 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53012 deadline: 1681314959918, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:59,937 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=604 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/B/26e91f36a34942a18b30efa40787a349 2023-04-12 15:54:59,967 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:59,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53078 deadline: 1681314959967, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:59,968 DEBUG [Thread-1639] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8154 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4., hostname=jenkins-hbase4.apache.org,38905,1681314818739, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-12 15:54:59,969 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:54:59,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53040 deadline: 1681314959969, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:54:59,970 DEBUG [Thread-1643] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8156 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4., hostname=jenkins-hbase4.apache.org,38905,1681314818739, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-12 15:55:00,221 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:00,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 413 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53012 deadline: 1681314960220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:00,345 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=604 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/C/e8e70acffc4f417bb83dfc79382c52c6 2023-04-12 15:55:00,349 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/A/fd3be8fb70fa49a09669feaaa0a0b917 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/fd3be8fb70fa49a09669feaaa0a0b917 2023-04-12 15:55:00,352 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/fd3be8fb70fa49a09669feaaa0a0b917, entries=200, sequenceid=604, filesize=14.3 K 2023-04-12 15:55:00,353 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/B/26e91f36a34942a18b30efa40787a349 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/26e91f36a34942a18b30efa40787a349 2023-04-12 15:55:00,356 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/26e91f36a34942a18b30efa40787a349, entries=150, sequenceid=604, filesize=11.9 K 2023-04-12 15:55:00,356 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/C/e8e70acffc4f417bb83dfc79382c52c6 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/e8e70acffc4f417bb83dfc79382c52c6 2023-04-12 15:55:00,360 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/e8e70acffc4f417bb83dfc79382c52c6, entries=150, sequenceid=604, filesize=11.9 K 2023-04-12 15:55:00,360 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for deda1ce51fef7ef8d26da255318c55e4 in 841ms, sequenceid=604, compaction requested=true 2023-04-12 15:55:00,361 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for deda1ce51fef7ef8d26da255318c55e4: 2023-04-12 15:55:00,361 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:55:00,361 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:55:00,361 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 15:55:00,361 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:55:00,361 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 15:55:00,362 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40122 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 15:55:00,362 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37682 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 15:55:00,362 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1912): deda1ce51fef7ef8d26da255318c55e4/A is initiating minor compaction (all files) 2023-04-12 15:55:00,362 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1912): deda1ce51fef7ef8d26da255318c55e4/B is initiating minor compaction (all files) 2023-04-12 15:55:00,362 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of deda1ce51fef7ef8d26da255318c55e4/A in TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:55:00,362 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2259): Starting compaction of deda1ce51fef7ef8d26da255318c55e4/B in TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:55:00,362 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/235f1d9d21b449cea481d1c5cccb2dd0, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/f58940cf4d934ab592fe1ad52b14b95b, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/fd3be8fb70fa49a09669feaaa0a0b917] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp, totalSize=39.2 K 2023-04-12 15:55:00,362 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/fb1a965435e64a5e9fc778f407e579fc, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/91bdd2217398440890aa16cc4d15f358, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/26e91f36a34942a18b30efa40787a349] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp, totalSize=36.8 K 2023-04-12 15:55:00,362 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 235f1d9d21b449cea481d1c5cccb2dd0, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=550, earliestPutTs=1681314898244 2023-04-12 15:55:00,362 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting fb1a965435e64a5e9fc778f407e579fc, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=550, earliestPutTs=1681314898244 2023-04-12 15:55:00,363 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting f58940cf4d934ab592fe1ad52b14b95b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=590, earliestPutTs=1681314898376 2023-04-12 15:55:00,363 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 91bdd2217398440890aa16cc4d15f358, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=590, earliestPutTs=1681314898376 2023-04-12 15:55:00,363 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting fd3be8fb70fa49a09669feaaa0a0b917, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=604, earliestPutTs=1681314899510 2023-04-12 15:55:00,363 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 26e91f36a34942a18b30efa40787a349, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=604, earliestPutTs=1681314899510 2023-04-12 15:55:00,368 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] throttle.PressureAwareThroughputController(145): deda1ce51fef7ef8d26da255318c55e4#B#compaction#308 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:55:00,369 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] throttle.PressureAwareThroughputController(145): deda1ce51fef7ef8d26da255318c55e4#A#compaction#309 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:55:00,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on deda1ce51fef7ef8d26da255318c55e4 2023-04-12 15:55:00,727 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing deda1ce51fef7ef8d26da255318c55e4 3/3 column families, dataSize=140.89 KB heapSize=405.61 KB 2023-04-12 15:55:00,727 DEBUG [Thread-1645] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x087b3095 to 127.0.0.1:60154 2023-04-12 15:55:00,727 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK deda1ce51fef7ef8d26da255318c55e4, store=A 2023-04-12 15:55:00,727 DEBUG [Thread-1645] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 15:55:00,727 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-12 15:55:00,727 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK deda1ce51fef7ef8d26da255318c55e4, store=B 2023-04-12 15:55:00,727 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-12 15:55:00,728 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK deda1ce51fef7ef8d26da255318c55e4, store=C 2023-04-12 15:55:00,728 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-12 15:55:00,777 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/A/7910fb15c07247568d4f2aa86356213e as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/7910fb15c07247568d4f2aa86356213e 2023-04-12 15:55:00,777 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/B/5cf5daf4100348df8a777eb6984f8c9f as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/5cf5daf4100348df8a777eb6984f8c9f 2023-04-12 15:55:00,780 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in deda1ce51fef7ef8d26da255318c55e4/B of deda1ce51fef7ef8d26da255318c55e4 into 5cf5daf4100348df8a777eb6984f8c9f(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:55:00,780 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in deda1ce51fef7ef8d26da255318c55e4/A of deda1ce51fef7ef8d26da255318c55e4 into 7910fb15c07247568d4f2aa86356213e(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:55:00,780 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for deda1ce51fef7ef8d26da255318c55e4: 2023-04-12 15:55:00,780 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for deda1ce51fef7ef8d26da255318c55e4: 2023-04-12 15:55:00,780 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4., storeName=deda1ce51fef7ef8d26da255318c55e4/B, priority=13, startTime=1681314900361; duration=0sec 2023-04-12 15:55:00,781 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4., storeName=deda1ce51fef7ef8d26da255318c55e4/A, priority=13, startTime=1681314900361; duration=0sec 2023-04-12 15:55:00,781 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:55:00,781 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:55:00,781 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-12 15:55:00,782 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49800 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-12 15:55:00,782 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1912): deda1ce51fef7ef8d26da255318c55e4/C is initiating minor compaction (all files) 2023-04-12 15:55:00,782 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2259): Starting compaction of deda1ce51fef7ef8d26da255318c55e4/C in TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:55:00,782 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/163142a310894c1fa1bb9762d24ace76, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/24bdbaef4f404fd4aa27d4ab52ac22ff, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/204c40eee97149059460ab8b59b7de12, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/e8e70acffc4f417bb83dfc79382c52c6] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp, totalSize=48.6 K 2023-04-12 15:55:00,782 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 163142a310894c1fa1bb9762d24ace76, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=540, earliestPutTs=1681314897619 2023-04-12 15:55:00,782 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 24bdbaef4f404fd4aa27d4ab52ac22ff, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=550, earliestPutTs=1681314898244 2023-04-12 15:55:00,782 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 204c40eee97149059460ab8b59b7de12, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=590, earliestPutTs=1681314898376 2023-04-12 15:55:00,783 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting e8e70acffc4f417bb83dfc79382c52c6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=604, earliestPutTs=1681314899510 2023-04-12 15:55:00,788 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] throttle.PressureAwareThroughputController(145): deda1ce51fef7ef8d26da255318c55e4#C#compaction#311 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:55:01,134 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=642 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/A/ac536ad4f7dd4425a41379b322feb4d5 2023-04-12 15:55:01,195 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/C/28044df9d1864f808a844b9167b6bb5a as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/28044df9d1864f808a844b9167b6bb5a 2023-04-12 15:55:01,199 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in deda1ce51fef7ef8d26da255318c55e4/C of deda1ce51fef7ef8d26da255318c55e4 into 28044df9d1864f808a844b9167b6bb5a(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:55:01,199 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for deda1ce51fef7ef8d26da255318c55e4: 2023-04-12 15:55:01,199 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4., storeName=deda1ce51fef7ef8d26da255318c55e4/C, priority=12, startTime=1681314900361; duration=0sec 2023-04-12 15:55:01,199 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:55:01,543 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=642 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/B/c78935e445f8469c86ee4c761c42ccc4 2023-04-12 15:55:01,953 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=642 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/C/aa26322d02c94d598253b2a1c8c6b3eb 2023-04-12 15:55:01,957 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/A/ac536ad4f7dd4425a41379b322feb4d5 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/ac536ad4f7dd4425a41379b322feb4d5 2023-04-12 15:55:01,960 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/ac536ad4f7dd4425a41379b322feb4d5, entries=150, sequenceid=642, filesize=11.9 K 2023-04-12 15:55:01,961 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/B/c78935e445f8469c86ee4c761c42ccc4 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/c78935e445f8469c86ee4c761c42ccc4 2023-04-12 15:55:01,964 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/c78935e445f8469c86ee4c761c42ccc4, entries=150, sequenceid=642, filesize=11.9 K 2023-04-12 15:55:01,964 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/C/aa26322d02c94d598253b2a1c8c6b3eb as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/aa26322d02c94d598253b2a1c8c6b3eb 2023-04-12 15:55:01,968 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/aa26322d02c94d598253b2a1c8c6b3eb, entries=150, sequenceid=642, filesize=11.9 K 2023-04-12 15:55:01,968 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~405.56 KB/415296, currentSize=0 B/0 for deda1ce51fef7ef8d26da255318c55e4 in 1241ms, sequenceid=642, compaction requested=false 2023-04-12 15:55:01,968 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for deda1ce51fef7ef8d26da255318c55e4: 2023-04-12 15:55:08,896 DEBUG [Thread-1637] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3157d4df to 127.0.0.1:60154 2023-04-12 15:55:08,896 DEBUG [Thread-1637] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 15:55:08,932 DEBUG [Thread-1641] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6919b2a0 to 127.0.0.1:60154 2023-04-12 15:55:08,933 DEBUG [Thread-1641] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 15:55:10,011 DEBUG [Thread-1643] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2ce5573e to 127.0.0.1:60154 2023-04-12 15:55:10,012 DEBUG [Thread-1643] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 15:55:10,026 DEBUG [Thread-1639] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3cf5bdb7 to 127.0.0.1:60154 2023-04-12 15:55:10,026 DEBUG [Thread-1639] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 15:55:10,026 INFO [Listener at localhost/43583] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-12 15:55:10,026 INFO [Listener at localhost/43583] hbase.AcidGuaranteesTestTool(394): wrote 48 2023-04-12 15:55:10,026 INFO [Listener at localhost/43583] hbase.AcidGuaranteesTestTool(394): wrote 64 2023-04-12 15:55:10,026 INFO [Listener at localhost/43583] hbase.AcidGuaranteesTestTool(394): wrote 56 2023-04-12 15:55:10,026 INFO [Listener at localhost/43583] hbase.AcidGuaranteesTestTool(394): wrote 99 2023-04-12 15:55:10,026 INFO [Listener at localhost/43583] hbase.AcidGuaranteesTestTool(394): wrote 264 2023-04-12 15:55:10,026 INFO [Listener at localhost/43583] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-12 15:55:10,026 INFO [Listener at localhost/43583] hbase.AcidGuaranteesTestTool(398): read 5546 2023-04-12 15:55:10,026 INFO [Listener at localhost/43583] hbase.AcidGuaranteesTestTool(398): read 5639 2023-04-12 15:55:10,026 INFO [Listener at localhost/43583] hbase.AcidGuaranteesTestTool(398): read 5374 2023-04-12 15:55:10,026 INFO [Listener at localhost/43583] hbase.AcidGuaranteesTestTool(398): read 5533 2023-04-12 15:55:10,026 INFO [Listener at localhost/43583] hbase.AcidGuaranteesTestTool(398): read 5645 2023-04-12 15:55:10,026 INFO [Listener at localhost/43583] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-12 15:55:10,026 INFO [Listener at localhost/43583] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-12 15:55:10,026 DEBUG [Listener at localhost/43583] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5c124820 to 127.0.0.1:60154 2023-04-12 15:55:10,027 DEBUG [Listener at localhost/43583] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 15:55:10,027 INFO [Listener at localhost/43583] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-12 15:55:10,027 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-12 15:55:10,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure2.ProcedureExecutor(1029): Stored pid=31, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-12 15:55:10,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-12 15:55:10,030 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681314910030"}]},"ts":"1681314910030"} 2023-04-12 15:55:10,031 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-12 15:55:10,033 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-12 15:55:10,034 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=deda1ce51fef7ef8d26da255318c55e4, UNASSIGN}] 2023-04-12 15:55:10,035 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=deda1ce51fef7ef8d26da255318c55e4, UNASSIGN 2023-04-12 15:55:10,035 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=deda1ce51fef7ef8d26da255318c55e4, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:10,036 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681314910035"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681314910035"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681314910035"}]},"ts":"1681314910035"} 2023-04-12 15:55:10,036 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=32, state=RUNNABLE; CloseRegionProcedure deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739}] 2023-04-12 15:55:10,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-12 15:55:10,188 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close deda1ce51fef7ef8d26da255318c55e4 2023-04-12 15:55:10,188 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing deda1ce51fef7ef8d26da255318c55e4, disabling compactions & flushes 2023-04-12 15:55:10,188 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:55:10,188 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:55:10,188 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. after waiting 0 ms 2023-04-12 15:55:10,188 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:55:10,188 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing deda1ce51fef7ef8d26da255318c55e4 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-12 15:55:10,188 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK deda1ce51fef7ef8d26da255318c55e4, store=A 2023-04-12 15:55:10,189 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:10,189 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK deda1ce51fef7ef8d26da255318c55e4, store=B 2023-04-12 15:55:10,189 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:10,189 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK deda1ce51fef7ef8d26da255318c55e4, store=C 2023-04-12 15:55:10,189 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:10,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-12 15:55:10,595 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=652 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/A/f994c1457b0f48b0a83c36aaf99a9ce6 2023-04-12 15:55:10,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-12 15:55:11,007 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=652 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/B/71ae5e20d0f04b8090e35ca7fa8e3da8 2023-04-12 15:55:11,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-12 15:55:11,415 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=652 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/C/efe9fb55fc554713a92eb8f322853370 2023-04-12 15:55:11,419 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/A/f994c1457b0f48b0a83c36aaf99a9ce6 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/f994c1457b0f48b0a83c36aaf99a9ce6 2023-04-12 15:55:11,422 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/f994c1457b0f48b0a83c36aaf99a9ce6, entries=150, sequenceid=652, filesize=11.9 K 2023-04-12 15:55:11,422 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/B/71ae5e20d0f04b8090e35ca7fa8e3da8 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/71ae5e20d0f04b8090e35ca7fa8e3da8 2023-04-12 15:55:11,425 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/71ae5e20d0f04b8090e35ca7fa8e3da8, entries=150, sequenceid=652, filesize=11.9 K 2023-04-12 15:55:11,426 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/.tmp/C/efe9fb55fc554713a92eb8f322853370 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/efe9fb55fc554713a92eb8f322853370 2023-04-12 15:55:11,429 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/efe9fb55fc554713a92eb8f322853370, entries=150, sequenceid=652, filesize=11.9 K 2023-04-12 15:55:11,429 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for deda1ce51fef7ef8d26da255318c55e4 in 1241ms, sequenceid=652, compaction requested=true 2023-04-12 15:55:11,449 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/b99771cc50114253849cbfb2756ffd83, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/386394a2b715447892aee082948007ac, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/9ac3c8a644cc4147bdd1e1a75edd0021, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/05aa95731e504028a8a31f024a0756e3, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/45de1a82700642a5b0231e12f3276cb9, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/212f2c600b0a45ae8c36202a72f0fc40, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/4073ebac96454afe80a6812184fff87b, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/09fff26ac3c640a4aa119d81e2b362ca, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/0152f039c6014feda7c02c0d81ee5d65, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/719ca553c87d49ae9ef9e7f4a0507870, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/993f63e50c5846f69f90d9a04a376d3a, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/7295be13e3ff4465866d021d82f31e76, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/b654ff237113487d8e5a691b7c9c0d94, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/fc27fa1ef4de4c7abf36e7c61dd079e3, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/f02f623dc7ef4913a53b0f3c8bcded12, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/622e1d87c7084859b72cd6c4fada624a, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/83a439098e854e14bbedf4374ebb533e, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/901263c6a73b48309269f92b6c0602f5, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/3dd684c350e34cb98a64f66f18f950b0, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/f185d5def0e840c89b7cbab5ec98c899, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/5a4f7d456d4748ec8a5c97c6a3ddcfec, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/1f0c597b4e18471da07895fa82095be9, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/0aa0cf31ecfb4f3989e97428c9edfde9, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/7082019db147440fa332ddaada47ebd7, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/0010282370924d608bbf17e6606abc93, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/b76e0fd08e50492eb7b5afe7b85fa74c, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/986287c66bc8405a8f5a2cf776e27c2b, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/69060ab698c841fab0c5f63b3b261569, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/be4c85cc009945c180a948cc04ff66e0, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/f33ae68e49164c1b84ded121bf43e366, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/bc97ef723aa34d9fb828dd5b598b6836, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/cabda8b29e3f43ef9132081cea19fefd, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/d5e62724e7d04940b0a163ceb0398854, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/235f1d9d21b449cea481d1c5cccb2dd0, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/f58940cf4d934ab592fe1ad52b14b95b, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/fd3be8fb70fa49a09669feaaa0a0b917] to archive 2023-04-12 15:55:11,450 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-12 15:55:11,451 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/b99771cc50114253849cbfb2756ffd83 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/b99771cc50114253849cbfb2756ffd83 2023-04-12 15:55:11,452 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/386394a2b715447892aee082948007ac to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/386394a2b715447892aee082948007ac 2023-04-12 15:55:11,453 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/9ac3c8a644cc4147bdd1e1a75edd0021 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/9ac3c8a644cc4147bdd1e1a75edd0021 2023-04-12 15:55:11,454 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/05aa95731e504028a8a31f024a0756e3 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/05aa95731e504028a8a31f024a0756e3 2023-04-12 15:55:11,455 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/45de1a82700642a5b0231e12f3276cb9 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/45de1a82700642a5b0231e12f3276cb9 2023-04-12 15:55:11,456 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/212f2c600b0a45ae8c36202a72f0fc40 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/212f2c600b0a45ae8c36202a72f0fc40 2023-04-12 15:55:11,457 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/4073ebac96454afe80a6812184fff87b to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/4073ebac96454afe80a6812184fff87b 2023-04-12 15:55:11,457 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/09fff26ac3c640a4aa119d81e2b362ca to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/09fff26ac3c640a4aa119d81e2b362ca 2023-04-12 15:55:11,458 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/0152f039c6014feda7c02c0d81ee5d65 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/0152f039c6014feda7c02c0d81ee5d65 2023-04-12 15:55:11,459 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/719ca553c87d49ae9ef9e7f4a0507870 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/719ca553c87d49ae9ef9e7f4a0507870 2023-04-12 15:55:11,460 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/993f63e50c5846f69f90d9a04a376d3a to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/993f63e50c5846f69f90d9a04a376d3a 2023-04-12 15:55:11,461 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/7295be13e3ff4465866d021d82f31e76 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/7295be13e3ff4465866d021d82f31e76 2023-04-12 15:55:11,462 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/b654ff237113487d8e5a691b7c9c0d94 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/b654ff237113487d8e5a691b7c9c0d94 2023-04-12 15:55:11,463 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/fc27fa1ef4de4c7abf36e7c61dd079e3 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/fc27fa1ef4de4c7abf36e7c61dd079e3 2023-04-12 15:55:11,464 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/f02f623dc7ef4913a53b0f3c8bcded12 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/f02f623dc7ef4913a53b0f3c8bcded12 2023-04-12 15:55:11,464 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/622e1d87c7084859b72cd6c4fada624a to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/622e1d87c7084859b72cd6c4fada624a 2023-04-12 15:55:11,465 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/83a439098e854e14bbedf4374ebb533e to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/83a439098e854e14bbedf4374ebb533e 2023-04-12 15:55:11,466 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/901263c6a73b48309269f92b6c0602f5 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/901263c6a73b48309269f92b6c0602f5 2023-04-12 15:55:11,467 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/3dd684c350e34cb98a64f66f18f950b0 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/3dd684c350e34cb98a64f66f18f950b0 2023-04-12 15:55:11,467 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/f185d5def0e840c89b7cbab5ec98c899 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/f185d5def0e840c89b7cbab5ec98c899 2023-04-12 15:55:11,468 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/5a4f7d456d4748ec8a5c97c6a3ddcfec to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/5a4f7d456d4748ec8a5c97c6a3ddcfec 2023-04-12 15:55:11,469 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/1f0c597b4e18471da07895fa82095be9 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/1f0c597b4e18471da07895fa82095be9 2023-04-12 15:55:11,470 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/0aa0cf31ecfb4f3989e97428c9edfde9 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/0aa0cf31ecfb4f3989e97428c9edfde9 2023-04-12 15:55:11,470 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/7082019db147440fa332ddaada47ebd7 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/7082019db147440fa332ddaada47ebd7 2023-04-12 15:55:11,471 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/0010282370924d608bbf17e6606abc93 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/0010282370924d608bbf17e6606abc93 2023-04-12 15:55:11,472 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/b76e0fd08e50492eb7b5afe7b85fa74c to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/b76e0fd08e50492eb7b5afe7b85fa74c 2023-04-12 15:55:11,473 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/986287c66bc8405a8f5a2cf776e27c2b to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/986287c66bc8405a8f5a2cf776e27c2b 2023-04-12 15:55:11,474 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/69060ab698c841fab0c5f63b3b261569 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/69060ab698c841fab0c5f63b3b261569 2023-04-12 15:55:11,474 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/be4c85cc009945c180a948cc04ff66e0 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/be4c85cc009945c180a948cc04ff66e0 2023-04-12 15:55:11,475 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/f33ae68e49164c1b84ded121bf43e366 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/f33ae68e49164c1b84ded121bf43e366 2023-04-12 15:55:11,476 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/bc97ef723aa34d9fb828dd5b598b6836 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/bc97ef723aa34d9fb828dd5b598b6836 2023-04-12 15:55:11,477 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/cabda8b29e3f43ef9132081cea19fefd to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/cabda8b29e3f43ef9132081cea19fefd 2023-04-12 15:55:11,478 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/d5e62724e7d04940b0a163ceb0398854 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/d5e62724e7d04940b0a163ceb0398854 2023-04-12 15:55:11,479 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/235f1d9d21b449cea481d1c5cccb2dd0 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/235f1d9d21b449cea481d1c5cccb2dd0 2023-04-12 15:55:11,480 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/f58940cf4d934ab592fe1ad52b14b95b to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/f58940cf4d934ab592fe1ad52b14b95b 2023-04-12 15:55:11,481 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/fd3be8fb70fa49a09669feaaa0a0b917 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/fd3be8fb70fa49a09669feaaa0a0b917 2023-04-12 15:55:11,484 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/a2523d426d53476cae46976432ba478f, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/c868446dfeb749948cabcda3c1f9c271, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/690c69166d0c44b2ba302fa0c8dce93f, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/c01ff948b9e242bb828790d2f8b52ac5, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/869399d589004bb1bec558eee7c12205, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/78ec40d786a84b8abae595ab823c553e, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/4ed251fa736c44c9b114969f480d818b, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/c5f406c8d1da42aa88d83ecccb8d1335, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/5d0fc1423a1e42bca92a105b87c6266e, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/9a066fa7382641d1ac5d8d9609b9ca03, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/ee3a5f35d43045a6a41a8a2f4611b3ca, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/27cc961916fc42658d3d2f3d0be19021, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/cf1523cb7d924a47b293492713c6b0cc, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/bd59478b2ea34208b7f43f15c90f57fe, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/0bb7d3bc29a04316978f7260fb78b780, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/408dec5f6a6a4bc3b657548399504c21, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/32c926e62d6f4400b6a6d14109b5b287, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/f5a0f2a16ee04a6f9e61f8dc971ce7d5, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/208bc335880a4650812d288c97195690, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/ca2496e21c8a4e6692f2780a74db43f0, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/5d1b9254dda7434eae6c18809769e973, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/6b0fa9de8eb242899db3db3d969dff39, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/d4a7809cb7e64e7da0639b8aeca65e7c, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/7603c8a2eb4a4341b21b81aba27eea17, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/65fb7b97edba4ca88b4b69faf2b89d73, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/e242b7a5ebf7415e9661fc10574b4075, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/6f13f9cc00884607abb5418d7a5c234c, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/d978b91e475f436db70e31596c9b65d9, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/0b26020e1a0f49a3ba814c3b71cac446, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/12d580cd11eb4fca8eb8f1b48d2c6e4c, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/6bbb97651c264e0b8fcc9d7a7d513ee0, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/bcea047f6e7b43c99aa9a532533a45a6, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/fb1a965435e64a5e9fc778f407e579fc, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/215b3e22f23a4074a2167e2e3c192bf6, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/91bdd2217398440890aa16cc4d15f358, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/26e91f36a34942a18b30efa40787a349] to archive 2023-04-12 15:55:11,484 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-12 15:55:11,485 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/a2523d426d53476cae46976432ba478f to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/a2523d426d53476cae46976432ba478f 2023-04-12 15:55:11,486 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/c868446dfeb749948cabcda3c1f9c271 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/c868446dfeb749948cabcda3c1f9c271 2023-04-12 15:55:11,487 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/690c69166d0c44b2ba302fa0c8dce93f to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/690c69166d0c44b2ba302fa0c8dce93f 2023-04-12 15:55:11,487 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/c01ff948b9e242bb828790d2f8b52ac5 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/c01ff948b9e242bb828790d2f8b52ac5 2023-04-12 15:55:11,488 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/869399d589004bb1bec558eee7c12205 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/869399d589004bb1bec558eee7c12205 2023-04-12 15:55:11,489 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/78ec40d786a84b8abae595ab823c553e to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/78ec40d786a84b8abae595ab823c553e 2023-04-12 15:55:11,490 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/4ed251fa736c44c9b114969f480d818b to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/4ed251fa736c44c9b114969f480d818b 2023-04-12 15:55:11,491 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/c5f406c8d1da42aa88d83ecccb8d1335 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/c5f406c8d1da42aa88d83ecccb8d1335 2023-04-12 15:55:11,491 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/5d0fc1423a1e42bca92a105b87c6266e to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/5d0fc1423a1e42bca92a105b87c6266e 2023-04-12 15:55:11,492 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/9a066fa7382641d1ac5d8d9609b9ca03 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/9a066fa7382641d1ac5d8d9609b9ca03 2023-04-12 15:55:11,493 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/ee3a5f35d43045a6a41a8a2f4611b3ca to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/ee3a5f35d43045a6a41a8a2f4611b3ca 2023-04-12 15:55:11,493 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/27cc961916fc42658d3d2f3d0be19021 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/27cc961916fc42658d3d2f3d0be19021 2023-04-12 15:55:11,494 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/cf1523cb7d924a47b293492713c6b0cc to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/cf1523cb7d924a47b293492713c6b0cc 2023-04-12 15:55:11,495 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/bd59478b2ea34208b7f43f15c90f57fe to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/bd59478b2ea34208b7f43f15c90f57fe 2023-04-12 15:55:11,495 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/0bb7d3bc29a04316978f7260fb78b780 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/0bb7d3bc29a04316978f7260fb78b780 2023-04-12 15:55:11,496 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/408dec5f6a6a4bc3b657548399504c21 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/408dec5f6a6a4bc3b657548399504c21 2023-04-12 15:55:11,497 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/32c926e62d6f4400b6a6d14109b5b287 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/32c926e62d6f4400b6a6d14109b5b287 2023-04-12 15:55:11,498 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/f5a0f2a16ee04a6f9e61f8dc971ce7d5 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/f5a0f2a16ee04a6f9e61f8dc971ce7d5 2023-04-12 15:55:11,498 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/208bc335880a4650812d288c97195690 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/208bc335880a4650812d288c97195690 2023-04-12 15:55:11,499 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/ca2496e21c8a4e6692f2780a74db43f0 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/ca2496e21c8a4e6692f2780a74db43f0 2023-04-12 15:55:11,500 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/5d1b9254dda7434eae6c18809769e973 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/5d1b9254dda7434eae6c18809769e973 2023-04-12 15:55:11,501 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/6b0fa9de8eb242899db3db3d969dff39 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/6b0fa9de8eb242899db3db3d969dff39 2023-04-12 15:55:11,502 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/d4a7809cb7e64e7da0639b8aeca65e7c to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/d4a7809cb7e64e7da0639b8aeca65e7c 2023-04-12 15:55:11,502 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/7603c8a2eb4a4341b21b81aba27eea17 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/7603c8a2eb4a4341b21b81aba27eea17 2023-04-12 15:55:11,503 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/65fb7b97edba4ca88b4b69faf2b89d73 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/65fb7b97edba4ca88b4b69faf2b89d73 2023-04-12 15:55:11,504 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/e242b7a5ebf7415e9661fc10574b4075 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/e242b7a5ebf7415e9661fc10574b4075 2023-04-12 15:55:11,505 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/6f13f9cc00884607abb5418d7a5c234c to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/6f13f9cc00884607abb5418d7a5c234c 2023-04-12 15:55:11,506 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/d978b91e475f436db70e31596c9b65d9 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/d978b91e475f436db70e31596c9b65d9 2023-04-12 15:55:11,507 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/0b26020e1a0f49a3ba814c3b71cac446 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/0b26020e1a0f49a3ba814c3b71cac446 2023-04-12 15:55:11,508 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/12d580cd11eb4fca8eb8f1b48d2c6e4c to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/12d580cd11eb4fca8eb8f1b48d2c6e4c 2023-04-12 15:55:11,509 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/6bbb97651c264e0b8fcc9d7a7d513ee0 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/6bbb97651c264e0b8fcc9d7a7d513ee0 2023-04-12 15:55:11,510 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/bcea047f6e7b43c99aa9a532533a45a6 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/bcea047f6e7b43c99aa9a532533a45a6 2023-04-12 15:55:11,511 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/fb1a965435e64a5e9fc778f407e579fc to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/fb1a965435e64a5e9fc778f407e579fc 2023-04-12 15:55:11,513 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/215b3e22f23a4074a2167e2e3c192bf6 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/215b3e22f23a4074a2167e2e3c192bf6 2023-04-12 15:55:11,514 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/91bdd2217398440890aa16cc4d15f358 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/91bdd2217398440890aa16cc4d15f358 2023-04-12 15:55:11,515 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/26e91f36a34942a18b30efa40787a349 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/26e91f36a34942a18b30efa40787a349 2023-04-12 15:55:11,518 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/f00576ffc88f428cb64b448151eb504b, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/26c610ee0ad54ceb809f277f972104d4, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/9e703618cbd94eeda86c14a7ff44ae98, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/b2ab52c67a604d3ca27b2f62d5084af9, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/1f8ae434e8024e1cadd722956b5368bd, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/f1335b82a9124d7787242be2317595f3, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/b92af03fc80642d1bc59464f856b851a, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/5c94082418524e11a8b3abeff6e0977b, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/7dbb1a2863af44839f90798ab13b47f6, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/754391c76d23499b9475cc8a247ace5e, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/2183b7bbeff84a27b052a71db45e6432, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/d43f8ae378264d50b474ef7c7fbf544e, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/88bd2a2da3b040df9ea5948d729d9742, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/da462dd7b00c46b09995e94cfd2a9cf9, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/0e2cffc2d94e4c2b9682d16e62c759e1, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/7c790d6efe1640ab8f931b59aed17680, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/c13908f0ee9943d382d20e68abf2a984, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/acc425b0bdf54d729d7d7a448c996ac9, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/1ab3da9470f4491bae562852f0621a4d, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/42507508ca7546ccab2ca456c438dc5f, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/3f74406dcb054cd9a061f91be65afd11, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/ac2d9286b4ce498988daa32f1d665382, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/37aa5784eb6048898414c6a5fa66dd7a, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/acf4b06b154f4cd0be674aaa7727df3c, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/fd9beb5e5eea428e847978b6ba355347, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/fa57eaec78f4480ab427de0528ab895d, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/45afff1641b64118a1aefd2efb4c0a20, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/1d647b39904642e683f14de2ef3b913b, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/dbaf57c12f3e4b269df648a537986792, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/0d21740358114e759f24948a2a314b38, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/163142a310894c1fa1bb9762d24ace76, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/1c14fa96704c45b5abec6068164ef66c, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/24bdbaef4f404fd4aa27d4ab52ac22ff, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/204c40eee97149059460ab8b59b7de12, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/e8e70acffc4f417bb83dfc79382c52c6] to archive 2023-04-12 15:55:11,519 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-12 15:55:11,520 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/f00576ffc88f428cb64b448151eb504b to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/f00576ffc88f428cb64b448151eb504b 2023-04-12 15:55:11,521 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/26c610ee0ad54ceb809f277f972104d4 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/26c610ee0ad54ceb809f277f972104d4 2023-04-12 15:55:11,522 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/9e703618cbd94eeda86c14a7ff44ae98 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/9e703618cbd94eeda86c14a7ff44ae98 2023-04-12 15:55:11,523 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/b2ab52c67a604d3ca27b2f62d5084af9 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/b2ab52c67a604d3ca27b2f62d5084af9 2023-04-12 15:55:11,524 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/1f8ae434e8024e1cadd722956b5368bd to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/1f8ae434e8024e1cadd722956b5368bd 2023-04-12 15:55:11,525 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/f1335b82a9124d7787242be2317595f3 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/f1335b82a9124d7787242be2317595f3 2023-04-12 15:55:11,525 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/b92af03fc80642d1bc59464f856b851a to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/b92af03fc80642d1bc59464f856b851a 2023-04-12 15:55:11,526 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/5c94082418524e11a8b3abeff6e0977b to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/5c94082418524e11a8b3abeff6e0977b 2023-04-12 15:55:11,527 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/7dbb1a2863af44839f90798ab13b47f6 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/7dbb1a2863af44839f90798ab13b47f6 2023-04-12 15:55:11,528 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/754391c76d23499b9475cc8a247ace5e to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/754391c76d23499b9475cc8a247ace5e 2023-04-12 15:55:11,529 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/2183b7bbeff84a27b052a71db45e6432 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/2183b7bbeff84a27b052a71db45e6432 2023-04-12 15:55:11,530 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/d43f8ae378264d50b474ef7c7fbf544e to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/d43f8ae378264d50b474ef7c7fbf544e 2023-04-12 15:55:11,530 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/88bd2a2da3b040df9ea5948d729d9742 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/88bd2a2da3b040df9ea5948d729d9742 2023-04-12 15:55:11,531 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/da462dd7b00c46b09995e94cfd2a9cf9 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/da462dd7b00c46b09995e94cfd2a9cf9 2023-04-12 15:55:11,532 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/0e2cffc2d94e4c2b9682d16e62c759e1 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/0e2cffc2d94e4c2b9682d16e62c759e1 2023-04-12 15:55:11,533 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/7c790d6efe1640ab8f931b59aed17680 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/7c790d6efe1640ab8f931b59aed17680 2023-04-12 15:55:11,534 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/c13908f0ee9943d382d20e68abf2a984 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/c13908f0ee9943d382d20e68abf2a984 2023-04-12 15:55:11,534 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/acc425b0bdf54d729d7d7a448c996ac9 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/acc425b0bdf54d729d7d7a448c996ac9 2023-04-12 15:55:11,535 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/1ab3da9470f4491bae562852f0621a4d to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/1ab3da9470f4491bae562852f0621a4d 2023-04-12 15:55:11,536 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/42507508ca7546ccab2ca456c438dc5f to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/42507508ca7546ccab2ca456c438dc5f 2023-04-12 15:55:11,537 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/3f74406dcb054cd9a061f91be65afd11 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/3f74406dcb054cd9a061f91be65afd11 2023-04-12 15:55:11,538 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/ac2d9286b4ce498988daa32f1d665382 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/ac2d9286b4ce498988daa32f1d665382 2023-04-12 15:55:11,539 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/37aa5784eb6048898414c6a5fa66dd7a to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/37aa5784eb6048898414c6a5fa66dd7a 2023-04-12 15:55:11,540 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/acf4b06b154f4cd0be674aaa7727df3c to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/acf4b06b154f4cd0be674aaa7727df3c 2023-04-12 15:55:11,540 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/fd9beb5e5eea428e847978b6ba355347 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/fd9beb5e5eea428e847978b6ba355347 2023-04-12 15:55:11,541 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/fa57eaec78f4480ab427de0528ab895d to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/fa57eaec78f4480ab427de0528ab895d 2023-04-12 15:55:11,542 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/45afff1641b64118a1aefd2efb4c0a20 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/45afff1641b64118a1aefd2efb4c0a20 2023-04-12 15:55:11,543 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/1d647b39904642e683f14de2ef3b913b to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/1d647b39904642e683f14de2ef3b913b 2023-04-12 15:55:11,544 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/dbaf57c12f3e4b269df648a537986792 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/dbaf57c12f3e4b269df648a537986792 2023-04-12 15:55:11,545 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/0d21740358114e759f24948a2a314b38 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/0d21740358114e759f24948a2a314b38 2023-04-12 15:55:11,545 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/163142a310894c1fa1bb9762d24ace76 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/163142a310894c1fa1bb9762d24ace76 2023-04-12 15:55:11,546 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/1c14fa96704c45b5abec6068164ef66c to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/1c14fa96704c45b5abec6068164ef66c 2023-04-12 15:55:11,547 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/24bdbaef4f404fd4aa27d4ab52ac22ff to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/24bdbaef4f404fd4aa27d4ab52ac22ff 2023-04-12 15:55:11,548 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/204c40eee97149059460ab8b59b7de12 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/204c40eee97149059460ab8b59b7de12 2023-04-12 15:55:11,549 DEBUG [StoreCloser-TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/e8e70acffc4f417bb83dfc79382c52c6 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/e8e70acffc4f417bb83dfc79382c52c6 2023-04-12 15:55:11,553 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/recovered.edits/655.seqid, newMaxSeqId=655, maxSeqId=1 2023-04-12 15:55:11,554 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4. 2023-04-12 15:55:11,554 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for deda1ce51fef7ef8d26da255318c55e4: 2023-04-12 15:55:11,555 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed deda1ce51fef7ef8d26da255318c55e4 2023-04-12 15:55:11,556 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=deda1ce51fef7ef8d26da255318c55e4, regionState=CLOSED 2023-04-12 15:55:11,556 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681314911555"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681314911555"}]},"ts":"1681314911555"} 2023-04-12 15:55:11,557 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=32 2023-04-12 15:55:11,557 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=32, state=SUCCESS; CloseRegionProcedure deda1ce51fef7ef8d26da255318c55e4, server=jenkins-hbase4.apache.org,38905,1681314818739 in 1.5200 sec 2023-04-12 15:55:11,559 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=32, resume processing ppid=31 2023-04-12 15:55:11,559 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=31, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=deda1ce51fef7ef8d26da255318c55e4, UNASSIGN in 1.5230 sec 2023-04-12 15:55:11,559 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681314911559"}]},"ts":"1681314911559"} 2023-04-12 15:55:11,560 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-12 15:55:11,561 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-12 15:55:11,562 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=31, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.5350 sec 2023-04-12 15:55:12,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-12 15:55:12,134 INFO [Listener at localhost/43583] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 31 completed 2023-04-12 15:55:12,134 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-12 15:55:12,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure2.ProcedureExecutor(1029): Stored pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-12 15:55:12,136 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-12 15:55:12,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-04-12 15:55:12,137 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=34, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-12 15:55:12,140 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4 2023-04-12 15:55:12,142 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A, FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B, FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C, FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/recovered.edits] 2023-04-12 15:55:12,144 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/7910fb15c07247568d4f2aa86356213e to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/7910fb15c07247568d4f2aa86356213e 2023-04-12 15:55:12,145 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/ac536ad4f7dd4425a41379b322feb4d5 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/ac536ad4f7dd4425a41379b322feb4d5 2023-04-12 15:55:12,146 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/f994c1457b0f48b0a83c36aaf99a9ce6 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/A/f994c1457b0f48b0a83c36aaf99a9ce6 2023-04-12 15:55:12,148 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/5cf5daf4100348df8a777eb6984f8c9f to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/5cf5daf4100348df8a777eb6984f8c9f 2023-04-12 15:55:12,149 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/71ae5e20d0f04b8090e35ca7fa8e3da8 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/71ae5e20d0f04b8090e35ca7fa8e3da8 2023-04-12 15:55:12,150 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/c78935e445f8469c86ee4c761c42ccc4 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/B/c78935e445f8469c86ee4c761c42ccc4 2023-04-12 15:55:12,152 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/28044df9d1864f808a844b9167b6bb5a to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/28044df9d1864f808a844b9167b6bb5a 2023-04-12 15:55:12,153 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/aa26322d02c94d598253b2a1c8c6b3eb to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/aa26322d02c94d598253b2a1c8c6b3eb 2023-04-12 15:55:12,154 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/efe9fb55fc554713a92eb8f322853370 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/C/efe9fb55fc554713a92eb8f322853370 2023-04-12 15:55:12,156 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/recovered.edits/655.seqid to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4/recovered.edits/655.seqid 2023-04-12 15:55:12,157 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/deda1ce51fef7ef8d26da255318c55e4 2023-04-12 15:55:12,157 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-12 15:55:12,158 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=34, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-12 15:55:12,159 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-12 15:55:12,161 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-12 15:55:12,162 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=34, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-12 15:55:12,162 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-12 15:55:12,162 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681314912162"}]},"ts":"9223372036854775807"} 2023-04-12 15:55:12,163 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-12 15:55:12,163 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => deda1ce51fef7ef8d26da255318c55e4, NAME => 'TestAcidGuarantees,,1681314877508.deda1ce51fef7ef8d26da255318c55e4.', STARTKEY => '', ENDKEY => ''}] 2023-04-12 15:55:12,163 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-12 15:55:12,163 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681314912163"}]},"ts":"9223372036854775807"} 2023-04-12 15:55:12,164 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-12 15:55:12,166 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=34, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-12 15:55:12,166 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=34, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 32 msec 2023-04-12 15:55:12,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-04-12 15:55:12,238 INFO [Listener at localhost/43583] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 34 completed 2023-04-12 15:55:12,249 INFO [Listener at localhost/43583] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=224 (was 231), OpenFileDescriptor=534 (was 570), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=356 (was 437), ProcessCount=169 (was 169), AvailableMemoryMB=524 (was 558) 2023-04-12 15:55:12,260 INFO [Listener at localhost/43583] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=224, OpenFileDescriptor=534, MaxFileDescriptor=60000, SystemLoadAverage=356, ProcessCount=169, AvailableMemoryMB=523 2023-04-12 15:55:12,262 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-12 15:55:12,262 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-12 15:55:12,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure2.ProcedureExecutor(1029): Stored pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-12 15:55:12,264 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-12 15:55:12,264 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 35 2023-04-12 15:55:12,265 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-12 15:55:12,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-12 15:55:12,266 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a 2023-04-12 15:55:12,266 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a empty. 2023-04-12 15:55:12,266 DEBUG [HFileArchiver-8] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a 2023-04-12 15:55:12,267 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-12 15:55:12,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-12 15:55:12,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-12 15:55:12,674 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-12 15:55:12,675 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => d7d8c2ad266b62b02ebd2f4b9a678b1a, NAME => 'TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp 2023-04-12 15:55:12,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-12 15:55:13,081 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-12 15:55:13,081 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing d7d8c2ad266b62b02ebd2f4b9a678b1a, disabling compactions & flushes 2023-04-12 15:55:13,081 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:13,081 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:13,081 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. after waiting 0 ms 2023-04-12 15:55:13,081 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:13,081 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:13,081 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for d7d8c2ad266b62b02ebd2f4b9a678b1a: 2023-04-12 15:55:13,083 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-12 15:55:13,084 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681314913084"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681314913084"}]},"ts":"1681314913084"} 2023-04-12 15:55:13,085 INFO [PEWorker-4] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-12 15:55:13,085 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-12 15:55:13,085 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681314913085"}]},"ts":"1681314913085"} 2023-04-12 15:55:13,086 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-12 15:55:13,090 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=d7d8c2ad266b62b02ebd2f4b9a678b1a, ASSIGN}] 2023-04-12 15:55:13,091 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=d7d8c2ad266b62b02ebd2f4b9a678b1a, ASSIGN 2023-04-12 15:55:13,091 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=d7d8c2ad266b62b02ebd2f4b9a678b1a, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,38905,1681314818739; forceNewPlan=false, retain=false 2023-04-12 15:55:13,242 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=d7d8c2ad266b62b02ebd2f4b9a678b1a, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:13,243 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681314913242"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681314913242"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681314913242"}]},"ts":"1681314913242"} 2023-04-12 15:55:13,244 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=37, ppid=36, state=RUNNABLE; OpenRegionProcedure d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739}] 2023-04-12 15:55:13,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-12 15:55:13,399 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:13,399 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => d7d8c2ad266b62b02ebd2f4b9a678b1a, NAME => 'TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a.', STARTKEY => '', ENDKEY => ''} 2023-04-12 15:55:13,399 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees d7d8c2ad266b62b02ebd2f4b9a678b1a 2023-04-12 15:55:13,399 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-12 15:55:13,399 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for d7d8c2ad266b62b02ebd2f4b9a678b1a 2023-04-12 15:55:13,399 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for d7d8c2ad266b62b02ebd2f4b9a678b1a 2023-04-12 15:55:13,400 INFO [StoreOpener-d7d8c2ad266b62b02ebd2f4b9a678b1a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region d7d8c2ad266b62b02ebd2f4b9a678b1a 2023-04-12 15:55:13,401 DEBUG [StoreOpener-d7d8c2ad266b62b02ebd2f4b9a678b1a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A 2023-04-12 15:55:13,401 DEBUG [StoreOpener-d7d8c2ad266b62b02ebd2f4b9a678b1a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A 2023-04-12 15:55:13,401 INFO [StoreOpener-d7d8c2ad266b62b02ebd2f4b9a678b1a-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-12 15:55:13,402 INFO [StoreOpener-d7d8c2ad266b62b02ebd2f4b9a678b1a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d7d8c2ad266b62b02ebd2f4b9a678b1a columnFamilyName A 2023-04-12 15:55:13,402 INFO [StoreOpener-d7d8c2ad266b62b02ebd2f4b9a678b1a-1] regionserver.HStore(310): Store=d7d8c2ad266b62b02ebd2f4b9a678b1a/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 15:55:13,402 INFO [StoreOpener-d7d8c2ad266b62b02ebd2f4b9a678b1a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region d7d8c2ad266b62b02ebd2f4b9a678b1a 2023-04-12 15:55:13,403 DEBUG [StoreOpener-d7d8c2ad266b62b02ebd2f4b9a678b1a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B 2023-04-12 15:55:13,403 DEBUG [StoreOpener-d7d8c2ad266b62b02ebd2f4b9a678b1a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B 2023-04-12 15:55:13,403 INFO [StoreOpener-d7d8c2ad266b62b02ebd2f4b9a678b1a-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-12 15:55:13,404 INFO [StoreOpener-d7d8c2ad266b62b02ebd2f4b9a678b1a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d7d8c2ad266b62b02ebd2f4b9a678b1a columnFamilyName B 2023-04-12 15:55:13,404 INFO [StoreOpener-d7d8c2ad266b62b02ebd2f4b9a678b1a-1] regionserver.HStore(310): Store=d7d8c2ad266b62b02ebd2f4b9a678b1a/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 15:55:13,404 INFO [StoreOpener-d7d8c2ad266b62b02ebd2f4b9a678b1a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region d7d8c2ad266b62b02ebd2f4b9a678b1a 2023-04-12 15:55:13,405 DEBUG [StoreOpener-d7d8c2ad266b62b02ebd2f4b9a678b1a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C 2023-04-12 15:55:13,405 DEBUG [StoreOpener-d7d8c2ad266b62b02ebd2f4b9a678b1a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C 2023-04-12 15:55:13,405 INFO [StoreOpener-d7d8c2ad266b62b02ebd2f4b9a678b1a-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-12 15:55:13,405 INFO [StoreOpener-d7d8c2ad266b62b02ebd2f4b9a678b1a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d7d8c2ad266b62b02ebd2f4b9a678b1a columnFamilyName C 2023-04-12 15:55:13,405 INFO [StoreOpener-d7d8c2ad266b62b02ebd2f4b9a678b1a-1] regionserver.HStore(310): Store=d7d8c2ad266b62b02ebd2f4b9a678b1a/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 15:55:13,406 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:13,406 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a 2023-04-12 15:55:13,406 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a 2023-04-12 15:55:13,408 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-12 15:55:13,409 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for d7d8c2ad266b62b02ebd2f4b9a678b1a 2023-04-12 15:55:13,410 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-12 15:55:13,411 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened d7d8c2ad266b62b02ebd2f4b9a678b1a; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65069616, jitterRate=-0.030387163162231445}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-12 15:55:13,411 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for d7d8c2ad266b62b02ebd2f4b9a678b1a: 2023-04-12 15:55:13,411 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a., pid=37, masterSystemTime=1681314913396 2023-04-12 15:55:13,412 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:13,413 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:13,413 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=d7d8c2ad266b62b02ebd2f4b9a678b1a, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:13,413 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681314913413"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681314913413"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681314913413"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681314913413"}]},"ts":"1681314913413"} 2023-04-12 15:55:13,415 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=37, resume processing ppid=36 2023-04-12 15:55:13,415 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=37, ppid=36, state=SUCCESS; OpenRegionProcedure d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 in 170 msec 2023-04-12 15:55:13,416 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=36, resume processing ppid=35 2023-04-12 15:55:13,416 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=36, ppid=35, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=d7d8c2ad266b62b02ebd2f4b9a678b1a, ASSIGN in 325 msec 2023-04-12 15:55:13,416 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-12 15:55:13,416 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681314913416"}]},"ts":"1681314913416"} 2023-04-12 15:55:13,417 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-12 15:55:13,419 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-12 15:55:13,420 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=35, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1570 sec 2023-04-12 15:55:14,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-12 15:55:14,369 INFO [Listener at localhost/43583] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 35 completed 2023-04-12 15:55:14,369 DEBUG [Listener at localhost/43583] zookeeper.ReadOnlyZKClient(139): Connect 0x0bcfc1e0 to 127.0.0.1:60154 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 15:55:14,374 DEBUG [Listener at localhost/43583] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@445919fd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 15:55:14,375 DEBUG [Listener at localhost/43583] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 15:55:14,377 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59786, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 15:55:14,378 DEBUG [Listener at localhost/43583] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-12 15:55:14,379 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37546, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-12 15:55:14,380 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-12 15:55:14,381 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-12 15:55:14,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure2.ProcedureExecutor(1029): Stored pid=38, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-12 15:55:14,790 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-12 15:55:14,791 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-12 15:55:14,791 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-12 15:55:14,793 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=39, ppid=38, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-12 15:55:14,795 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=d7d8c2ad266b62b02ebd2f4b9a678b1a, REOPEN/MOVE}] 2023-04-12 15:55:14,795 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=d7d8c2ad266b62b02ebd2f4b9a678b1a, REOPEN/MOVE 2023-04-12 15:55:14,796 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=d7d8c2ad266b62b02ebd2f4b9a678b1a, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:14,796 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681314914796"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681314914796"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681314914796"}]},"ts":"1681314914796"} 2023-04-12 15:55:14,797 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=41, ppid=40, state=RUNNABLE; CloseRegionProcedure d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739}] 2023-04-12 15:55:14,948 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close d7d8c2ad266b62b02ebd2f4b9a678b1a 2023-04-12 15:55:14,949 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing d7d8c2ad266b62b02ebd2f4b9a678b1a, disabling compactions & flushes 2023-04-12 15:55:14,949 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:14,949 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:14,949 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. after waiting 0 ms 2023-04-12 15:55:14,949 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:14,952 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-12 15:55:14,953 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:14,953 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for d7d8c2ad266b62b02ebd2f4b9a678b1a: 2023-04-12 15:55:14,953 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: d7d8c2ad266b62b02ebd2f4b9a678b1a to self. 2023-04-12 15:55:14,954 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed d7d8c2ad266b62b02ebd2f4b9a678b1a 2023-04-12 15:55:14,954 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=d7d8c2ad266b62b02ebd2f4b9a678b1a, regionState=CLOSED 2023-04-12 15:55:14,954 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681314914954"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681314914954"}]},"ts":"1681314914954"} 2023-04-12 15:55:14,956 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=41, resume processing ppid=40 2023-04-12 15:55:14,956 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=41, ppid=40, state=SUCCESS; CloseRegionProcedure d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 in 158 msec 2023-04-12 15:55:14,956 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=d7d8c2ad266b62b02ebd2f4b9a678b1a, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,38905,1681314818739; forceNewPlan=false, retain=true 2023-04-12 15:55:15,107 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=d7d8c2ad266b62b02ebd2f4b9a678b1a, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:15,107 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681314915107"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681314915107"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681314915107"}]},"ts":"1681314915107"} 2023-04-12 15:55:15,108 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=42, ppid=40, state=RUNNABLE; OpenRegionProcedure d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739}] 2023-04-12 15:55:15,261 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:15,261 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => d7d8c2ad266b62b02ebd2f4b9a678b1a, NAME => 'TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a.', STARTKEY => '', ENDKEY => ''} 2023-04-12 15:55:15,262 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees d7d8c2ad266b62b02ebd2f4b9a678b1a 2023-04-12 15:55:15,262 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-12 15:55:15,262 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for d7d8c2ad266b62b02ebd2f4b9a678b1a 2023-04-12 15:55:15,262 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for d7d8c2ad266b62b02ebd2f4b9a678b1a 2023-04-12 15:55:15,263 INFO [StoreOpener-d7d8c2ad266b62b02ebd2f4b9a678b1a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region d7d8c2ad266b62b02ebd2f4b9a678b1a 2023-04-12 15:55:15,264 DEBUG [StoreOpener-d7d8c2ad266b62b02ebd2f4b9a678b1a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A 2023-04-12 15:55:15,264 DEBUG [StoreOpener-d7d8c2ad266b62b02ebd2f4b9a678b1a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A 2023-04-12 15:55:15,264 INFO [StoreOpener-d7d8c2ad266b62b02ebd2f4b9a678b1a-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-12 15:55:15,264 INFO [StoreOpener-d7d8c2ad266b62b02ebd2f4b9a678b1a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d7d8c2ad266b62b02ebd2f4b9a678b1a columnFamilyName A 2023-04-12 15:55:15,266 INFO [StoreOpener-d7d8c2ad266b62b02ebd2f4b9a678b1a-1] regionserver.HStore(310): Store=d7d8c2ad266b62b02ebd2f4b9a678b1a/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 15:55:15,266 INFO [StoreOpener-d7d8c2ad266b62b02ebd2f4b9a678b1a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region d7d8c2ad266b62b02ebd2f4b9a678b1a 2023-04-12 15:55:15,266 DEBUG [StoreOpener-d7d8c2ad266b62b02ebd2f4b9a678b1a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B 2023-04-12 15:55:15,266 DEBUG [StoreOpener-d7d8c2ad266b62b02ebd2f4b9a678b1a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B 2023-04-12 15:55:15,267 INFO [StoreOpener-d7d8c2ad266b62b02ebd2f4b9a678b1a-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-12 15:55:15,267 INFO [StoreOpener-d7d8c2ad266b62b02ebd2f4b9a678b1a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d7d8c2ad266b62b02ebd2f4b9a678b1a columnFamilyName B 2023-04-12 15:55:15,267 INFO [StoreOpener-d7d8c2ad266b62b02ebd2f4b9a678b1a-1] regionserver.HStore(310): Store=d7d8c2ad266b62b02ebd2f4b9a678b1a/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 15:55:15,267 INFO [StoreOpener-d7d8c2ad266b62b02ebd2f4b9a678b1a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region d7d8c2ad266b62b02ebd2f4b9a678b1a 2023-04-12 15:55:15,268 DEBUG [StoreOpener-d7d8c2ad266b62b02ebd2f4b9a678b1a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C 2023-04-12 15:55:15,268 DEBUG [StoreOpener-d7d8c2ad266b62b02ebd2f4b9a678b1a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C 2023-04-12 15:55:15,268 INFO [StoreOpener-d7d8c2ad266b62b02ebd2f4b9a678b1a-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-12 15:55:15,268 INFO [StoreOpener-d7d8c2ad266b62b02ebd2f4b9a678b1a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d7d8c2ad266b62b02ebd2f4b9a678b1a columnFamilyName C 2023-04-12 15:55:15,268 INFO [StoreOpener-d7d8c2ad266b62b02ebd2f4b9a678b1a-1] regionserver.HStore(310): Store=d7d8c2ad266b62b02ebd2f4b9a678b1a/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 15:55:15,269 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:15,269 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a 2023-04-12 15:55:15,270 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a 2023-04-12 15:55:15,271 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-12 15:55:15,272 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for d7d8c2ad266b62b02ebd2f4b9a678b1a 2023-04-12 15:55:15,273 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened d7d8c2ad266b62b02ebd2f4b9a678b1a; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60006096, jitterRate=-0.10583949089050293}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-12 15:55:15,273 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for d7d8c2ad266b62b02ebd2f4b9a678b1a: 2023-04-12 15:55:15,273 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a., pid=42, masterSystemTime=1681314915259 2023-04-12 15:55:15,274 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:15,274 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:15,275 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=d7d8c2ad266b62b02ebd2f4b9a678b1a, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:15,275 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681314915275"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681314915275"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681314915275"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681314915275"}]},"ts":"1681314915275"} 2023-04-12 15:55:15,277 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=42, resume processing ppid=40 2023-04-12 15:55:15,277 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=42, ppid=40, state=SUCCESS; OpenRegionProcedure d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 in 168 msec 2023-04-12 15:55:15,278 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=40, resume processing ppid=39 2023-04-12 15:55:15,278 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=40, ppid=39, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=d7d8c2ad266b62b02ebd2f4b9a678b1a, REOPEN/MOVE in 482 msec 2023-04-12 15:55:15,279 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=39, resume processing ppid=38 2023-04-12 15:55:15,279 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=39, ppid=38, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 485 msec 2023-04-12 15:55:15,281 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=38, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 899 msec 2023-04-12 15:55:15,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1227): Checking to see if procedure is done pid=38 2023-04-12 15:55:15,282 DEBUG [Listener at localhost/43583] zookeeper.ReadOnlyZKClient(139): Connect 0x38341134 to 127.0.0.1:60154 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 15:55:15,286 DEBUG [Listener at localhost/43583] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4d010713, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 15:55:15,287 DEBUG [Listener at localhost/43583] zookeeper.ReadOnlyZKClient(139): Connect 0x71b5058b to 127.0.0.1:60154 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 15:55:15,290 DEBUG [Listener at localhost/43583] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@69100738, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 15:55:15,290 DEBUG [Listener at localhost/43583] zookeeper.ReadOnlyZKClient(139): Connect 0x3620ee35 to 127.0.0.1:60154 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 15:55:15,294 DEBUG [Listener at localhost/43583] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6f603c27, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 15:55:15,294 DEBUG [Listener at localhost/43583] zookeeper.ReadOnlyZKClient(139): Connect 0x4fe1db6a to 127.0.0.1:60154 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 15:55:15,298 DEBUG [Listener at localhost/43583] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7e01d5e5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 15:55:15,298 DEBUG [Listener at localhost/43583] zookeeper.ReadOnlyZKClient(139): Connect 0x10377b50 to 127.0.0.1:60154 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 15:55:15,302 DEBUG [Listener at localhost/43583] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@51ccd8b7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 15:55:15,302 DEBUG [Listener at localhost/43583] zookeeper.ReadOnlyZKClient(139): Connect 0x68a0e21a to 127.0.0.1:60154 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 15:55:15,305 DEBUG [Listener at localhost/43583] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5f9e86d9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 15:55:15,305 DEBUG [Listener at localhost/43583] zookeeper.ReadOnlyZKClient(139): Connect 0x1ec12b02 to 127.0.0.1:60154 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 15:55:15,308 DEBUG [Listener at localhost/43583] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@711ebaf2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 15:55:15,308 DEBUG [Listener at localhost/43583] zookeeper.ReadOnlyZKClient(139): Connect 0x1d7ba242 to 127.0.0.1:60154 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 15:55:15,314 DEBUG [Listener at localhost/43583] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@a7503d4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 15:55:15,314 DEBUG [Listener at localhost/43583] zookeeper.ReadOnlyZKClient(139): Connect 0x3e079da1 to 127.0.0.1:60154 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 15:55:15,318 DEBUG [Listener at localhost/43583] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7eae4191, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 15:55:15,318 DEBUG [Listener at localhost/43583] zookeeper.ReadOnlyZKClient(139): Connect 0x159c5e0c to 127.0.0.1:60154 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 15:55:15,321 DEBUG [Listener at localhost/43583] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1774338b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 15:55:15,325 DEBUG [hconnection-0x5651b708-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 15:55:15,326 DEBUG [hconnection-0x7d025e3a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 15:55:15,326 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59790, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 15:55:15,327 DEBUG [hconnection-0x1047fcaa-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 15:55:15,327 DEBUG [hconnection-0x5912456b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 15:55:15,328 DEBUG [hconnection-0x6d1b98e4-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 15:55:15,329 DEBUG [hconnection-0x479f5e24-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 15:55:15,330 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59804, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 15:55:15,330 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59824, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 15:55:15,330 DEBUG [hconnection-0x1db04254-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 15:55:15,330 DEBUG [hconnection-0x41d5ecdd-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 15:55:15,330 DEBUG [hconnection-0x40a2bd9b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 15:55:15,332 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59838, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 15:55:15,332 DEBUG [hconnection-0x2017cb4f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 15:55:15,333 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59810, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 15:55:15,333 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59868, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 15:55:15,336 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59852, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 15:55:15,338 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59892, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 15:55:15,338 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59882, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 15:55:15,338 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59902, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 15:55:15,340 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:15,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:15,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:15,341 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:15,342 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:15,342 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:15,342 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:15,342 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:15,342 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:15,343 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:15,343 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:15,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:15,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:15,344 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:15,344 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:15,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:15,346 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:15,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:15,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:15,347 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:15,347 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:15,347 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:15,348 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:15,348 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:15,348 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:15,352 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:15,352 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:15,352 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:15,353 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. started... 2023-04-12 15:55:15,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on d7d8c2ad266b62b02ebd2f4b9a678b1a 2023-04-12 15:55:15,353 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. as already flushing 2023-04-12 15:55:15,353 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d7d8c2ad266b62b02ebd2f4b9a678b1a 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-12 15:55:15,354 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7d8c2ad266b62b02ebd2f4b9a678b1a, store=A 2023-04-12 15:55:15,354 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:15,354 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7d8c2ad266b62b02ebd2f4b9a678b1a, store=B 2023-04-12 15:55:15,354 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:15,354 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7d8c2ad266b62b02ebd2f4b9a678b1a, store=C 2023-04-12 15:55:15,354 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:15,379 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681314915370, max timestamp=1681314915377Num uniques 859; 2023-04-12 15:55:15,379 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-12 15:55:15,379 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681314915370, max timestamp=1681314915377Num uniques 859; 2023-04-12 15:55:15,379 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681314915370, max timestamp=1681314915377Num uniques 859; 2023-04-12 15:55:15,380 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-12 15:55:15,380 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-12 15:55:15,396 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:15,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59882 deadline: 1681314975391, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:15,397 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:15,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59902 deadline: 1681314975392, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:15,401 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:15,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59804 deadline: 1681314975393, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:15,402 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:15,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59790 deadline: 1681314975394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:15,402 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:15,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59810 deadline: 1681314975394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:15,499 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:15,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59882 deadline: 1681314975497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:15,500 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:15,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59902 deadline: 1681314975498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:15,508 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:15,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59804 deadline: 1681314975502, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:15,508 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:15,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59790 deadline: 1681314975503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:15,509 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:15,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59810 deadline: 1681314975503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:15,703 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:15,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59902 deadline: 1681314975701, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:15,704 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:15,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59882 deadline: 1681314975701, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:15,711 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:15,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59804 deadline: 1681314975709, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:15,711 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:15,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59810 deadline: 1681314975709, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:15,715 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:15,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59790 deadline: 1681314975710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:15,781 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230412888ab3999aac48ae91965a5b98583879 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412888ab3999aac48ae91965a5b98583879 2023-04-12 15:55:16,007 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:16,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59882 deadline: 1681314976006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:16,008 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:16,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59902 deadline: 1681314976006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:16,016 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:16,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59804 deadline: 1681314976012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:16,016 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:16,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59810 deadline: 1681314976014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:16,021 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:16,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59790 deadline: 1681314976018, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:16,190 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/A/3d3c11c962474d92b3abc65caa0fcffe 2023-04-12 15:55:16,513 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:16,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59882 deadline: 1681314976510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:16,513 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:16,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59902 deadline: 1681314976510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:16,521 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:16,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59804 deadline: 1681314976520, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:16,524 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:16,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59810 deadline: 1681314976521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:16,529 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:16,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59790 deadline: 1681314976525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:16,602 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/B/e7dab56fe9474868a75f977b842924f8 2023-04-12 15:55:16,612 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/C/ba57ef6c23bd4c409856f688ab60c0f2 2023-04-12 15:55:16,616 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/A/3d3c11c962474d92b3abc65caa0fcffe as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/3d3c11c962474d92b3abc65caa0fcffe 2023-04-12 15:55:16,620 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/3d3c11c962474d92b3abc65caa0fcffe, entries=150, sequenceid=15, filesize=25.1 K 2023-04-12 15:55:16,621 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/B/e7dab56fe9474868a75f977b842924f8 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/e7dab56fe9474868a75f977b842924f8 2023-04-12 15:55:16,628 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/e7dab56fe9474868a75f977b842924f8, entries=150, sequenceid=15, filesize=11.6 K 2023-04-12 15:55:16,629 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/C/ba57ef6c23bd4c409856f688ab60c0f2 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/ba57ef6c23bd4c409856f688ab60c0f2 2023-04-12 15:55:16,639 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/ba57ef6c23bd4c409856f688ab60c0f2, entries=150, sequenceid=15, filesize=11.6 K 2023-04-12 15:55:16,639 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.StoreScanner(961): StoreScanner already closing. There is no need to updateReaders 2023-04-12 15:55:16,640 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for d7d8c2ad266b62b02ebd2f4b9a678b1a in 1287ms, sequenceid=15, compaction requested=false 2023-04-12 15:55:16,640 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d7d8c2ad266b62b02ebd2f4b9a678b1a: 2023-04-12 15:55:16,640 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:16,640 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:16,641 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:16,641 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:16,641 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:16,641 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:16,644 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:16,644 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:16,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:16,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:16,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:16,644 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:16,644 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:16,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:16,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:16,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:16,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:16,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:16,646 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:16,646 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53e940c[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:16,646 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:16,646 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:16,647 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:16,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:16,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:16,647 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:16,648 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:16,648 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:16,648 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:16,648 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:16,651 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:16,651 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:16,651 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:16,651 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:16,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:16,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:16,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:16,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:16,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:16,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:16,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:16,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:16,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:16,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:16,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:16,654 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:16,654 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:16,654 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:16,654 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:16,654 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:16,654 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:16,655 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:16,656 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:16,655 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:16,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:16,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:16,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:16,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:16,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:16,656 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:16,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:16,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:16,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:16,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:16,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:16,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:16,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:16,658 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:16,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:16,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:16,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:16,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:16,664 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:16,664 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:16,664 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:16,664 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:16,664 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:16,664 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:16,664 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:16,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:16,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:16,664 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:16,664 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:16,664 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:16,664 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:16,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:16,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:16,664 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:16,664 DEBUG [Thread-2432] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:16,665 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:16,665 DEBUG [Thread-2432] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:16,765 DEBUG [Thread-2432] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:16,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:16,767 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:16,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:16,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:16,768 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:16,768 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:16,768 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:16,768 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:16,769 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:16,769 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:16,770 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:16,770 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:16,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:16,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:16,770 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:16,770 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:16,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:16,771 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:16,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:16,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:16,771 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:16,771 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:16,771 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:16,771 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:16,771 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:16,771 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:16,771 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:16,771 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:16,771 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:16,772 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. started... 2023-04-12 15:55:16,772 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d7d8c2ad266b62b02ebd2f4b9a678b1a 3/3 column families, dataSize=127.47 KB heapSize=355.69 KB 2023-04-12 15:55:16,772 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7d8c2ad266b62b02ebd2f4b9a678b1a, store=A 2023-04-12 15:55:16,772 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:55:16,772 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7d8c2ad266b62b02ebd2f4b9a678b1a, store=B 2023-04-12 15:55:16,772 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:55:16,772 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7d8c2ad266b62b02ebd2f4b9a678b1a, store=C 2023-04-12 15:55:16,772 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:55:16,791 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304127690e4d3f9b742889f52a8880c9d541d to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304127690e4d3f9b742889f52a8880c9d541d 2023-04-12 15:55:17,088 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-12 15:55:17,198 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=49, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/A/30e2dd705f1a4a58a737ef2d51ce87b8 2023-04-12 15:55:17,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on d7d8c2ad266b62b02ebd2f4b9a678b1a 2023-04-12 15:55:17,517 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. as already flushing 2023-04-12 15:55:17,536 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:17,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59882 deadline: 1681314977530, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:17,536 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:17,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59804 deadline: 1681314977530, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:17,540 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:17,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59810 deadline: 1681314977535, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:17,542 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:17,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59902 deadline: 1681314977535, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:17,546 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:17,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59790 deadline: 1681314977537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:17,608 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=49 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/B/013430c6da3a44b787d28110def3551f 2023-04-12 15:55:17,638 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:17,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59882 deadline: 1681314977637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:17,638 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:17,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59804 deadline: 1681314977637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:17,644 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:17,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59810 deadline: 1681314977641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:17,644 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:17,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59902 deadline: 1681314977643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:17,842 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:17,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59882 deadline: 1681314977839, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:17,842 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:17,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59804 deadline: 1681314977839, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:17,848 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:17,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59810 deadline: 1681314977845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:17,848 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:17,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59902 deadline: 1681314977846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,020 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=49 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/C/093dce0e9ed9469a83276e7d9a30ecc5 2023-04-12 15:55:18,024 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/A/30e2dd705f1a4a58a737ef2d51ce87b8 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/30e2dd705f1a4a58a737ef2d51ce87b8 2023-04-12 15:55:18,027 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/30e2dd705f1a4a58a737ef2d51ce87b8, entries=150, sequenceid=49, filesize=25.1 K 2023-04-12 15:55:18,028 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/B/013430c6da3a44b787d28110def3551f as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/013430c6da3a44b787d28110def3551f 2023-04-12 15:55:18,031 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/013430c6da3a44b787d28110def3551f, entries=150, sequenceid=49, filesize=11.6 K 2023-04-12 15:55:18,032 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/C/093dce0e9ed9469a83276e7d9a30ecc5 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/093dce0e9ed9469a83276e7d9a30ecc5 2023-04-12 15:55:18,038 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/093dce0e9ed9469a83276e7d9a30ecc5, entries=150, sequenceid=49, filesize=11.6 K 2023-04-12 15:55:18,039 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~355.64 KB/364176, currentSize=53.67 KB/54960 for d7d8c2ad266b62b02ebd2f4b9a678b1a in 1267ms, sequenceid=49, compaction requested=false 2023-04-12 15:55:18,039 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d7d8c2ad266b62b02ebd2f4b9a678b1a: 2023-04-12 15:55:18,039 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:18,039 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:18,041 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:18,041 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,041 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:18,041 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:18,043 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:18,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:18,043 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:18,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:18,043 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:18,043 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:18,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:18,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:18,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:18,045 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:18,045 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1cb9ecfc[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:18,045 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:18,045 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:18,046 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,046 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:18,046 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:18,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:18,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:18,047 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:18,047 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:18,047 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:18,049 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,049 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:18,049 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:18,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:18,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:18,049 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:18,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:18,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:18,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:18,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:18,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:18,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:18,052 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:18,052 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:18,052 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:18,052 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:18,052 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:18,052 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:18,053 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:18,053 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:18,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:18,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:18,053 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:18,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:18,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:18,054 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:18,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:18,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:18,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:18,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:18,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:18,055 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:18,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:18,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:18,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:18,058 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,058 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:18,058 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:18,058 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:18,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:18,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:18,058 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,058 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:18,058 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:18,058 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:18,058 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:18,058 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:18,058 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:18,058 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:18,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:18,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:18,058 DEBUG [Thread-2432] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:18,058 DEBUG [Thread-2432] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:18,059 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:18,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on d7d8c2ad266b62b02ebd2f4b9a678b1a 2023-04-12 15:55:18,146 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d7d8c2ad266b62b02ebd2f4b9a678b1a 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-12 15:55:18,147 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7d8c2ad266b62b02ebd2f4b9a678b1a, store=A 2023-04-12 15:55:18,147 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:18,147 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7d8c2ad266b62b02ebd2f4b9a678b1a, store=B 2023-04-12 15:55:18,147 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:18,147 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7d8c2ad266b62b02ebd2f4b9a678b1a, store=C 2023-04-12 15:55:18,147 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:18,159 DEBUG [Thread-2432] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:18,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:18,161 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:18,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:18,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:18,162 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:18,162 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:18,162 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:18,162 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:18,162 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:18,162 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:18,164 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:18,164 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:18,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:18,164 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,164 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:18,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:18,165 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:18,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:18,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:18,165 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:18,165 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:18,165 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:18,166 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:18,166 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:18,166 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:18,166 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:18,166 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:18,166 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:18,166 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. started... 2023-04-12 15:55:18,166 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. as already flushing 2023-04-12 15:55:18,166 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:18,166 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:18,166 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:18,166 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,166 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:18,166 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:18,169 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,169 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:18,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:18,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:18,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:18,169 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:18,170 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:18,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:18,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:18,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:18,171 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:18,171 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b279edc[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:18,171 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:18,171 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:18,172 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:18,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:18,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:18,173 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:18,173 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:18,173 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:18,173 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,173 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:18,174 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,174 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:18,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:18,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:18,174 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:18,174 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:18,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:18,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:18,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:18,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:18,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:18,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:18,176 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:18,176 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:18,176 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:18,176 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:18,176 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:18,176 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:18,178 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:18,178 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:18,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:18,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:18,179 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:18,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:18,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:18,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:18,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:18,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:18,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:18,180 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:18,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:18,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:18,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:18,181 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:18,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:18,182 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:18,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59902 deadline: 1681314978172, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,184 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,184 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:18,184 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:18,184 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,184 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:18,184 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:18,184 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:18,184 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:18,184 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:18,184 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:18,184 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:18,184 DEBUG [Thread-2432] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:18,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:18,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:18,184 DEBUG [Thread-2432] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:18,184 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:18,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:18,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:18,185 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:18,185 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:18,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59882 deadline: 1681314978178, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,185 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:18,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59804 deadline: 1681314978178, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,186 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:18,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59810 deadline: 1681314978179, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,285 DEBUG [Thread-2432] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:18,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:18,287 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:18,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:18,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:18,288 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:18,288 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:18,288 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:18,288 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:18,288 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:18,288 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:18,290 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,290 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:18,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:18,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:18,290 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:18,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59902 deadline: 1681314978284, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,290 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,290 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:18,290 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:18,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:18,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59882 deadline: 1681314978286, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,290 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:18,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59804 deadline: 1681314978286, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,291 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:18,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:18,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:18,291 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:18,291 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:18,291 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:18,291 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:18,291 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:18,291 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:18,292 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:18,292 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:18,292 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:18,292 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. started... 2023-04-12 15:55:18,292 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. as already flushing 2023-04-12 15:55:18,292 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:18,292 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:18,292 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:18,292 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,292 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:18,292 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:18,293 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,293 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:18,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:18,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:18,294 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:18,294 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:18,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:18,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:18,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:18,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:18,295 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:18,295 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b24b2fe[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:18,295 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:18,295 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:18,296 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:18,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:18,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:18,296 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:18,296 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:18,296 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:18,296 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,296 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:18,296 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:18,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59810 deadline: 1681314978289, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,298 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:18,298 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:18,298 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,298 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:18,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:18,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:18,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:18,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:18,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:18,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:18,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:18,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:18,301 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:18,301 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:18,301 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:18,301 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:18,301 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:18,301 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:18,303 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:18,303 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:18,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:18,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:18,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:18,303 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:18,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:18,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:18,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:18,303 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:18,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:18,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:18,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:18,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:18,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:18,306 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:18,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:18,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,310 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,310 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:18,310 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:18,310 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:18,310 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:18,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:18,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:18,310 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:18,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:18,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:18,310 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:18,310 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,310 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:18,310 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:18,310 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:18,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:18,310 DEBUG [Thread-2432] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:18,310 DEBUG [Thread-2432] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:18,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:18,410 DEBUG [Thread-2432] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:18,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:18,413 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:18,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:18,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:18,414 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:18,414 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:18,414 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:18,414 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:18,414 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:18,415 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:18,417 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:18,417 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:18,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:18,417 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,417 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:18,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:18,417 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:18,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:18,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:18,418 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:18,418 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:18,418 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:18,418 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:18,418 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:18,418 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:18,418 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:18,418 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:18,418 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:18,418 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. started... 2023-04-12 15:55:18,418 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. as already flushing 2023-04-12 15:55:18,418 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:18,419 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:18,419 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:18,419 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,419 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:18,419 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:18,420 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,420 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:18,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:18,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:18,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:18,421 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:18,421 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:18,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:18,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:18,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:18,422 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:18,422 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a4bc962[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:18,422 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:18,422 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:18,423 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:18,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:18,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:18,423 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:18,423 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:18,423 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:18,423 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,423 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:18,425 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:18,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:18,425 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:18,425 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:18,425 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:18,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:18,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:18,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:18,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:18,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:18,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:18,427 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:18,427 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:18,427 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:18,427 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:18,427 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:18,427 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:18,430 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:18,430 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:18,430 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:18,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:18,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:18,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:18,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:18,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:18,430 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:18,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:18,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:18,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:18,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:18,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:18,431 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:18,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:18,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:18,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,433 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,433 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:18,433 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:18,433 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:18,433 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:18,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:18,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:18,434 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:18,434 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:18,434 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:18,434 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,434 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:18,434 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:18,434 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:18,434 DEBUG [Thread-2432] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:18,434 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:18,434 DEBUG [Thread-2432] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:18,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:18,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:18,494 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:18,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59882 deadline: 1681314978491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,494 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:18,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59902 deadline: 1681314978492, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,494 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:18,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59804 deadline: 1681314978492, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,502 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:18,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59810 deadline: 1681314978499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,534 DEBUG [Thread-2432] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:18,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:18,536 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:18,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:18,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:18,537 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:18,538 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:18,538 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:18,538 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:18,538 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:18,538 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:18,539 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:18,539 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:18,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:18,540 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,540 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:18,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:18,543 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:18,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:18,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:18,543 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:18,544 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:18,544 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:18,544 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:18,544 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:18,544 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:18,544 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:18,544 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:18,544 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:18,544 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. started... 2023-04-12 15:55:18,544 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. as already flushing 2023-04-12 15:55:18,544 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:18,544 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:18,544 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:18,544 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,544 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:18,544 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:18,548 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:18,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:18,548 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:18,550 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:18,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:18,550 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:18,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:18,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:18,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:18,551 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:18,551 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29c614db[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:18,551 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:18,551 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:18,554 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:18,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:18,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:18,554 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:18,554 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,554 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:18,554 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:18,554 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:18,556 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:18,556 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:18,556 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,556 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:18,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:18,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:18,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:18,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:18,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:18,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:18,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:18,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:18,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:18,558 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:18,559 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:18,559 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:18,559 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:18,559 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:18,561 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:18,561 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:18,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:18,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:18,561 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:18,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:18,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:18,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:18,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:18,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:18,562 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:18,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:18,562 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304126694cb1707be417bb75f5a9005c175d1 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304126694cb1707be417bb75f5a9005c175d1 2023-04-12 15:55:18,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:18,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:18,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:18,563 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:18,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:18,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,568 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=62, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/A/a1365fc1f5df41c19d4e0a660c14d627 2023-04-12 15:55:18,569 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:18,569 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,569 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:18,569 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:18,569 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:18,569 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:18,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:18,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:18,569 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:18,569 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,569 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:18,569 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:18,569 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:18,569 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:18,569 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:18,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:18,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:18,570 DEBUG [Thread-2432] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:18,570 DEBUG [Thread-2432] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:18,670 DEBUG [Thread-2432] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:18,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:18,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:18,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:18,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:18,674 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:18,674 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:18,674 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:18,674 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:18,675 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:18,675 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:18,677 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,677 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:18,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:18,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:18,677 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,677 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:18,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:18,678 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:18,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:18,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:18,678 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:18,679 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:18,679 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:18,679 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:18,679 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:18,679 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:18,679 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:18,679 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:18,679 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:18,680 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. started... 2023-04-12 15:55:18,680 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. as already flushing 2023-04-12 15:55:18,680 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:18,680 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:18,680 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:18,680 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,680 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:18,680 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:18,681 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,681 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:18,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:18,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:18,681 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:18,681 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:18,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:18,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:18,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:18,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:18,682 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:18,683 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ee0be41[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:18,683 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:18,683 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:18,684 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:18,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:18,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:18,684 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:18,684 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:18,684 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:18,684 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,684 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:18,685 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:18,685 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,685 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:18,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:18,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:18,685 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:18,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:18,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:18,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:18,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:18,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:18,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:18,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:18,688 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:18,688 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:18,688 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:18,688 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:18,688 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:18,691 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:18,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:18,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:18,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:18,691 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:18,691 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:18,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:18,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:18,692 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:18,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:18,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:18,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:18,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:18,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:18,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:18,693 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:18,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:18,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,695 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,695 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:18,695 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:18,695 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:18,695 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:18,695 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:18,695 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:18,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:18,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:18,695 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:18,695 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,696 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:18,696 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:18,696 DEBUG [Thread-2432] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:18,696 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:18,696 DEBUG [Thread-2432] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:18,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:18,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:18,796 DEBUG [Thread-2432] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:18,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:18,798 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:18,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59882 deadline: 1681314978796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,798 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:18,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:18,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:18,799 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:18,799 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:18,799 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:18,799 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:18,800 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:18,800 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:18,800 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:18,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59902 deadline: 1681314978797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,801 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:18,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59804 deadline: 1681314978797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,803 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,803 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:18,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:18,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:18,804 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,804 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:18,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:18,804 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:18,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:18,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:18,804 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:18,805 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:18,805 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:18,805 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:18,805 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:18,805 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:18,805 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:18,805 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:18,805 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:18,805 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. started... 2023-04-12 15:55:18,805 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. as already flushing 2023-04-12 15:55:18,805 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:18,805 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:18,805 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:18,805 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,805 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:18,805 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:18,806 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:18,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59810 deadline: 1681314978803, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,813 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,813 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:18,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:18,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:18,813 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:18,813 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:18,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:18,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:18,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:18,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:18,814 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:18,814 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70fd5421[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:18,814 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:18,815 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:18,816 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:18,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:18,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:18,816 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:18,816 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:18,816 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:18,816 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,816 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:18,818 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,818 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:18,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:18,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:18,818 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:18,818 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:18,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:18,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:18,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:18,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:18,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:18,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:18,822 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:18,822 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:18,823 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:18,823 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:18,823 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:18,823 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:18,824 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:18,824 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:18,824 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:18,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:18,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:18,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:18,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:18,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:18,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:18,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:18,824 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:18,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:18,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:18,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:18,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:18,825 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:18,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:18,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,829 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,829 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:18,829 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:18,829 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:18,829 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:18,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:18,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:18,829 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:18,829 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:18,829 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:18,829 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:18,829 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,829 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:18,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:18,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:18,829 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:18,829 DEBUG [Thread-2432] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:18,829 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:18,829 DEBUG [Thread-2432] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:18,929 DEBUG [Thread-2432] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:18,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:18,931 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:18,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:18,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:18,933 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:18,933 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:18,933 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:18,933 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:18,933 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:18,933 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:18,934 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,934 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:18,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:18,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:18,934 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,934 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:18,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:18,935 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:18,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:18,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:18,935 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:18,935 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:18,936 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:18,936 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:18,936 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:18,936 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:18,936 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:18,936 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:18,936 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:18,936 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. started... 2023-04-12 15:55:18,936 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. as already flushing 2023-04-12 15:55:18,936 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:18,936 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:18,936 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:18,936 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,936 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:18,936 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:18,937 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,938 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:18,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:18,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:18,938 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:18,938 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:18,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:18,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:18,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:18,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:18,939 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:18,939 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54d98385[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:18,939 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:18,939 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:18,940 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:18,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:18,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:18,940 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,940 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:18,940 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:18,940 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:18,940 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:18,942 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,942 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:18,942 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:18,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:18,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:18,942 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:18,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:18,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:18,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:18,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:18,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:18,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:18,945 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:18,945 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:18,945 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:18,945 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:18,945 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:18,945 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:18,946 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:18,946 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:18,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:18,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:18,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:18,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:18,946 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:18,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:18,946 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:18,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:18,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:18,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:18,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:18,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:18,947 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:18,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:18,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:18,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,949 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,949 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:18,949 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:18,949 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:18,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:18,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:18,950 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:18,950 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:18,950 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:18,950 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:18,950 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:18,950 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:18,950 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:18,950 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:18,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:18,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:18,950 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:18,950 DEBUG [Thread-2432] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:18,950 DEBUG [Thread-2432] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:18,984 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=62 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/B/dfeaba1ef6924392b2ccbe0ed259afe8 2023-04-12 15:55:19,050 DEBUG [Thread-2432] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:19,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:19,053 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:19,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:19,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:19,054 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:19,054 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:19,054 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:19,054 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:19,054 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:19,054 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:19,055 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:19,055 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:19,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:19,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:19,056 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:19,056 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:19,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:19,056 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:19,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:19,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:19,057 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:19,057 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:19,057 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:19,057 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:19,057 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:19,057 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:19,057 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:19,057 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:19,057 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:19,057 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. started... 2023-04-12 15:55:19,057 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. as already flushing 2023-04-12 15:55:19,057 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:19,057 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:19,057 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:19,058 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:19,058 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:19,058 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:19,060 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:19,060 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:19,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:19,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:19,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:19,060 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:19,060 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:19,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:19,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:19,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:19,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:19,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:19,062 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:19,062 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4da02509[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:19,062 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:19,062 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:19,063 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:19,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:19,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:19,063 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:19,063 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:19,063 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:19,063 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:19,063 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:19,064 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:19,064 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:19,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:19,064 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:19,064 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:19,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:19,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:19,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:19,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:19,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:19,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:19,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:19,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:19,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:19,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:19,067 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:19,067 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:19,067 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:19,067 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:19,067 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:19,067 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:19,068 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:19,068 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:19,068 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:19,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:19,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:19,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:19,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:19,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:19,069 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:19,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:19,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:19,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:19,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:19,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:19,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:19,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:19,070 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:19,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:19,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:19,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:19,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:19,073 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:19,074 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:19,073 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:19,074 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:19,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:19,074 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:19,074 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:19,074 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:19,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:19,074 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:19,074 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:19,074 DEBUG [Thread-2432] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:19,074 DEBUG [Thread-2432] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:19,074 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:19,074 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:19,074 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:19,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:19,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:19,074 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:19,174 DEBUG [Thread-2432] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:19,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:19,176 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:19,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:19,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:19,177 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:19,177 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:19,177 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:19,177 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:19,177 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:19,178 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:19,179 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:19,179 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:19,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:19,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:19,179 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:19,179 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:19,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:19,180 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:19,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:19,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:19,180 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:19,180 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:19,180 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:19,181 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:19,181 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:19,181 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:19,181 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:19,181 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:19,181 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:19,181 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. started... 2023-04-12 15:55:19,181 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. as already flushing 2023-04-12 15:55:19,181 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:19,181 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:19,181 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:19,181 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:19,181 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:19,181 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:19,185 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:19,185 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:19,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:19,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:19,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:19,185 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:19,185 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:19,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:19,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:19,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:19,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:19,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:19,187 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:19,187 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1669884a[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:19,187 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:19,187 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:19,193 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:19,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:19,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:19,193 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:19,193 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:19,193 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:19,193 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:19,193 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:19,194 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:19,194 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:19,194 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:19,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:19,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:19,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:19,194 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:19,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:19,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:19,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:19,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:19,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:19,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:19,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:19,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:19,196 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:19,197 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:19,198 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:19,198 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:19,198 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:19,198 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:19,199 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:19,200 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:19,200 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:19,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:19,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:19,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:19,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:19,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:19,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:19,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:19,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:19,200 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:19,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:19,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:19,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:19,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:19,201 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:19,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:19,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:19,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:19,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:19,203 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:19,203 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:19,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:19,203 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:19,203 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:19,203 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:19,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:19,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:19,203 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:19,203 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:19,204 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:19,204 DEBUG [Thread-2432] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:19,204 DEBUG [Thread-2432] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:19,204 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:19,204 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:19,204 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:19,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:19,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:19,204 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:19,304 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:19,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59902 deadline: 1681314979301, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:19,304 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:19,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59882 deadline: 1681314979302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:19,304 DEBUG [Thread-2432] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:19,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:19,306 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:19,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:19,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:19,307 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:19,307 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:19,307 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:19,307 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:19,308 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:19,308 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:19,310 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:19,310 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:19,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:19,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:19,311 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:19,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59804 deadline: 1681314979305, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:19,311 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:19,311 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:19,311 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:19,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:19,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59810 deadline: 1681314979310, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:19,311 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:19,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:19,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:19,312 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:19,312 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:19,312 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:19,312 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:19,312 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:19,312 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:19,312 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:19,312 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:19,312 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:19,312 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. started... 2023-04-12 15:55:19,313 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. as already flushing 2023-04-12 15:55:19,313 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:19,313 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:19,313 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:19,313 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:19,313 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:19,313 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:19,314 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:19,314 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:19,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:19,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:19,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:19,314 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:19,315 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:19,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:19,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:19,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:19,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:19,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:19,316 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:19,316 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@caf9afa[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:19,316 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:19,316 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:19,317 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:19,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:19,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:19,317 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:19,317 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:19,317 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:19,317 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:19,317 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:19,318 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:19,318 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:19,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:19,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:19,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:19,318 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:19,318 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:19,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:19,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:19,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:19,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:19,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:19,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:19,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:19,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:19,320 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:19,320 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:19,321 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:19,321 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:19,321 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:19,321 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:19,323 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:19,323 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:19,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:19,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:19,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:19,323 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:19,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:19,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:19,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:19,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:19,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:19,324 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:19,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:19,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:19,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:19,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:19,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:19,325 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:19,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:19,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:19,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:19,327 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:19,327 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:19,327 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:19,328 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:19,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:19,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:19,328 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:19,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:19,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:19,328 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:19,328 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:19,328 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:19,328 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:19,328 DEBUG [Thread-2432] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:19,328 DEBUG [Thread-2432] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:19,328 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:19,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:19,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:19,394 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=62 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/C/29139bb214e14f14af370063cbe612ef 2023-04-12 15:55:19,398 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/A/a1365fc1f5df41c19d4e0a660c14d627 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/a1365fc1f5df41c19d4e0a660c14d627 2023-04-12 15:55:19,401 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/a1365fc1f5df41c19d4e0a660c14d627, entries=150, sequenceid=62, filesize=25.1 K 2023-04-12 15:55:19,402 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/B/dfeaba1ef6924392b2ccbe0ed259afe8 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/dfeaba1ef6924392b2ccbe0ed259afe8 2023-04-12 15:55:19,405 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/dfeaba1ef6924392b2ccbe0ed259afe8, entries=150, sequenceid=62, filesize=11.6 K 2023-04-12 15:55:19,409 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/C/29139bb214e14f14af370063cbe612ef as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/29139bb214e14f14af370063cbe612ef 2023-04-12 15:55:19,412 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/29139bb214e14f14af370063cbe612ef, entries=150, sequenceid=62, filesize=11.6 K 2023-04-12 15:55:19,413 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for d7d8c2ad266b62b02ebd2f4b9a678b1a in 1267ms, sequenceid=62, compaction requested=true 2023-04-12 15:55:19,413 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d7d8c2ad266b62b02ebd2f4b9a678b1a: 2023-04-12 15:55:19,413 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:55:19,413 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 15:55:19,413 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 15:55:19,413 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:55:19,413 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:55:19,416 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 77238 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 15:55:19,416 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 15:55:19,416 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1912): d7d8c2ad266b62b02ebd2f4b9a678b1a/A is initiating minor compaction (all files) 2023-04-12 15:55:19,417 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1912): d7d8c2ad266b62b02ebd2f4b9a678b1a/B is initiating minor compaction (all files) 2023-04-12 15:55:19,418 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d7d8c2ad266b62b02ebd2f4b9a678b1a/A in TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:19,418 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d7d8c2ad266b62b02ebd2f4b9a678b1a/B in TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:19,418 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/3d3c11c962474d92b3abc65caa0fcffe, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/30e2dd705f1a4a58a737ef2d51ce87b8, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/a1365fc1f5df41c19d4e0a660c14d627] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp, totalSize=75.4 K 2023-04-12 15:55:19,418 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/e7dab56fe9474868a75f977b842924f8, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/013430c6da3a44b787d28110def3551f, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/dfeaba1ef6924392b2ccbe0ed259afe8] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp, totalSize=34.8 K 2023-04-12 15:55:19,418 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 3d3c11c962474d92b3abc65caa0fcffe, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1681314915340 2023-04-12 15:55:19,419 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting e7dab56fe9474868a75f977b842924f8, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1681314915340 2023-04-12 15:55:19,419 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 30e2dd705f1a4a58a737ef2d51ce87b8, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1681314915391 2023-04-12 15:55:19,419 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 013430c6da3a44b787d28110def3551f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1681314915391 2023-04-12 15:55:19,419 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting a1365fc1f5df41c19d4e0a660c14d627, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1681314917529 2023-04-12 15:55:19,420 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting dfeaba1ef6924392b2ccbe0ed259afe8, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1681314917529 2023-04-12 15:55:19,427 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] throttle.PressureAwareThroughputController(145): d7d8c2ad266b62b02ebd2f4b9a678b1a#B#compaction#327 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:55:19,428 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] throttle.PressureAwareThroughputController(145): d7d8c2ad266b62b02ebd2f4b9a678b1a#A#compaction#326 average throughput is 6.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:55:19,428 DEBUG [Thread-2432] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:19,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:19,431 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:19,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:19,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:19,435 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:19,435 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:19,436 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:19,436 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:19,436 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:19,436 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:19,438 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:19,438 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:19,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:19,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:19,438 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:19,438 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:19,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:19,438 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:19,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:19,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:19,439 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:19,439 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:19,439 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:19,439 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:19,439 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:19,439 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:19,439 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:19,439 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:19,439 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:19,439 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. started... 2023-04-12 15:55:19,439 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d7d8c2ad266b62b02ebd2f4b9a678b1a 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-12 15:55:19,439 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7d8c2ad266b62b02ebd2f4b9a678b1a, store=A 2023-04-12 15:55:19,440 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:19,440 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7d8c2ad266b62b02ebd2f4b9a678b1a, store=B 2023-04-12 15:55:19,440 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:19,440 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7d8c2ad266b62b02ebd2f4b9a678b1a, store=C 2023-04-12 15:55:19,440 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:19,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on d7d8c2ad266b62b02ebd2f4b9a678b1a 2023-04-12 15:55:19,558 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. as already flushing 2023-04-12 15:55:19,607 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:19,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59790 deadline: 1681314979602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:19,710 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:19,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59790 deadline: 1681314979709, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:19,835 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/B/d9f56c4fd8294b0b81536f3321e977b4 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/d9f56c4fd8294b0b81536f3321e977b4 2023-04-12 15:55:19,841 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d7d8c2ad266b62b02ebd2f4b9a678b1a/B of d7d8c2ad266b62b02ebd2f4b9a678b1a into d9f56c4fd8294b0b81536f3321e977b4(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:55:19,842 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d7d8c2ad266b62b02ebd2f4b9a678b1a: 2023-04-12 15:55:19,842 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a., storeName=d7d8c2ad266b62b02ebd2f4b9a678b1a/B, priority=13, startTime=1681314919413; duration=0sec 2023-04-12 15:55:19,842 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:55:19,842 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 15:55:19,849 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 15:55:19,849 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1912): d7d8c2ad266b62b02ebd2f4b9a678b1a/C is initiating minor compaction (all files) 2023-04-12 15:55:19,849 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d7d8c2ad266b62b02ebd2f4b9a678b1a/C in TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:19,850 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/ba57ef6c23bd4c409856f688ab60c0f2, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/093dce0e9ed9469a83276e7d9a30ecc5, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/29139bb214e14f14af370063cbe612ef] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp, totalSize=34.8 K 2023-04-12 15:55:19,850 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting ba57ef6c23bd4c409856f688ab60c0f2, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1681314915340 2023-04-12 15:55:19,851 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 093dce0e9ed9469a83276e7d9a30ecc5, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1681314915391 2023-04-12 15:55:19,851 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 29139bb214e14f14af370063cbe612ef, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1681314917529 2023-04-12 15:55:19,858 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230412191db7a04cf04a36b29612928a589f6b to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412191db7a04cf04a36b29612928a589f6b 2023-04-12 15:55:19,859 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] throttle.PressureAwareThroughputController(145): d7d8c2ad266b62b02ebd2f4b9a678b1a#C#compaction#329 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:55:19,866 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=81, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/A/2b512416a6da4c3c8c4aee846f2ca3d7 2023-04-12 15:55:19,883 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=81 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/B/d31c50b168fc4069b43b9418ccdcf464 2023-04-12 15:55:19,914 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:19,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59790 deadline: 1681314979912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:20,219 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:20,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59790 deadline: 1681314980217, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:20,268 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/C/0de2554cf73244a8b34114410e9bd747 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/0de2554cf73244a8b34114410e9bd747 2023-04-12 15:55:20,271 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d7d8c2ad266b62b02ebd2f4b9a678b1a/C of d7d8c2ad266b62b02ebd2f4b9a678b1a into 0de2554cf73244a8b34114410e9bd747(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:55:20,271 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d7d8c2ad266b62b02ebd2f4b9a678b1a: 2023-04-12 15:55:20,272 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a., storeName=d7d8c2ad266b62b02ebd2f4b9a678b1a/C, priority=13, startTime=1681314919413; duration=0sec 2023-04-12 15:55:20,272 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:55:20,300 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=81 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/C/7c26a31a9e8540409e1fa547f90c6ce0 2023-04-12 15:55:20,304 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/A/2b512416a6da4c3c8c4aee846f2ca3d7 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/2b512416a6da4c3c8c4aee846f2ca3d7 2023-04-12 15:55:20,307 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/2b512416a6da4c3c8c4aee846f2ca3d7, entries=150, sequenceid=81, filesize=25.1 K 2023-04-12 15:55:20,308 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/B/d31c50b168fc4069b43b9418ccdcf464 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/d31c50b168fc4069b43b9418ccdcf464 2023-04-12 15:55:20,311 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:20,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59902 deadline: 1681314980308, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:20,312 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/d31c50b168fc4069b43b9418ccdcf464, entries=150, sequenceid=81, filesize=11.6 K 2023-04-12 15:55:20,312 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:20,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59882 deadline: 1681314980309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:20,313 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/C/7c26a31a9e8540409e1fa547f90c6ce0 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/7c26a31a9e8540409e1fa547f90c6ce0 2023-04-12 15:55:20,319 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/7c26a31a9e8540409e1fa547f90c6ce0, entries=150, sequenceid=81, filesize=11.6 K 2023-04-12 15:55:20,320 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for d7d8c2ad266b62b02ebd2f4b9a678b1a in 881ms, sequenceid=81, compaction requested=false 2023-04-12 15:55:20,320 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d7d8c2ad266b62b02ebd2f4b9a678b1a: 2023-04-12 15:55:20,320 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:20,320 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:20,320 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:20,320 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:20,320 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:20,320 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:20,322 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:20,322 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:20,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:20,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:20,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:20,322 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:20,323 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:20,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:20,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:20,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:20,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:20,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on d7d8c2ad266b62b02ebd2f4b9a678b1a 2023-04-12 15:55:20,324 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d7d8c2ad266b62b02ebd2f4b9a678b1a 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-12 15:55:20,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:20,325 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:20,325 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14f2f541[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:20,325 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:20,325 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:20,325 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7d8c2ad266b62b02ebd2f4b9a678b1a, store=A 2023-04-12 15:55:20,325 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:20,325 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7d8c2ad266b62b02ebd2f4b9a678b1a, store=B 2023-04-12 15:55:20,325 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:20,326 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7d8c2ad266b62b02ebd2f4b9a678b1a, store=C 2023-04-12 15:55:20,326 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:20,326 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:20,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:20,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:20,326 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:20,326 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:20,326 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:20,327 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:20,327 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:20,328 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:20,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:20,328 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:20,328 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:20,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:20,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:20,328 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:20,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:20,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:20,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:20,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:20,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:20,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:20,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:20,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:20,334 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:20,334 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:20,334 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:20,334 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:20,334 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:20,334 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:20,338 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:20,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:20,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:20,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:20,338 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:20,338 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:20,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:20,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:20,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:20,339 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:20,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:20,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:20,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:20,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:20,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:20,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:20,340 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:20,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:20,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:20,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:20,342 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:20,342 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:20,342 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:20,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:20,342 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:20,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:20,342 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:20,342 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:20,342 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:20,342 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:20,342 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:20,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:20,342 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:20,342 DEBUG [Thread-2432] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:20,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:20,342 DEBUG [Thread-2432] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:20,343 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:20,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:20,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:20,379 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:20,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59804 deadline: 1681314980374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:20,389 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:20,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59810 deadline: 1681314980379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:20,443 DEBUG [Thread-2432] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:20,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:20,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:20,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:20,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:20,448 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:20,448 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:20,448 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:20,448 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:20,449 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:20,449 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:20,450 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:20,450 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:20,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:20,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:20,450 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:20,450 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:20,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:20,451 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:20,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:20,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:20,451 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:20,451 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:20,451 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:20,451 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:20,451 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:20,452 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:20,452 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:20,452 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:20,452 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. started... 2023-04-12 15:55:20,452 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. as already flushing 2023-04-12 15:55:20,452 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:20,452 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:20,452 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:20,452 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:20,452 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:20,452 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:20,456 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:20,456 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:20,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:20,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:20,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:20,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:20,456 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:20,456 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:20,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:20,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:20,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:20,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:20,458 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:20,458 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@583602bb[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:20,458 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:20,458 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:20,459 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:20,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:20,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:20,459 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:20,459 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:20,459 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:20,463 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:20,463 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:20,463 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:20,463 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:20,463 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:20,463 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:20,464 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:20,464 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:20,464 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:20,464 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:20,464 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:20,466 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:20,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:20,466 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:20,466 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:20,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:20,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:20,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:20,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:20,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:20,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:20,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:20,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:20,470 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:20,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:20,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:20,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:20,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:20,472 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:20,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:20,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:20,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:20,474 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:20,474 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:20,474 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:20,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:20,474 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:20,474 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:20,474 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:20,474 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:20,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:20,474 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:20,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:20,474 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:20,475 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:20,475 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:20,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:20,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:20,480 DEBUG [Thread-2432] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:20,480 DEBUG [Thread-2432] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:20,481 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:20,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59804 deadline: 1681314980480, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:20,496 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:20,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59810 deadline: 1681314980490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:20,580 DEBUG [Thread-2432] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:20,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:20,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:20,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:20,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:20,588 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:20,588 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:20,588 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:20,588 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:20,593 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:20,593 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:20,596 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:20,596 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:20,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:20,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:20,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:20,596 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:20,596 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:20,597 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:20,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:20,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:20,597 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:20,597 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:20,598 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:20,598 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:20,598 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:20,598 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:20,600 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:20,600 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:20,600 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:20,600 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. started... 2023-04-12 15:55:20,600 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. as already flushing 2023-04-12 15:55:20,600 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:20,600 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:20,600 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:20,600 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:20,600 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:20,600 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:20,602 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:20,602 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:20,602 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:20,602 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:20,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:20,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:20,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:20,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:20,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:20,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:20,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:20,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:20,605 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:20,605 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c7ea2f[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:20,605 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:20,605 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:20,607 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:20,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:20,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:20,608 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:20,608 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:20,608 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:20,609 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:20,610 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:20,610 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:20,610 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:20,610 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:20,610 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:20,610 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:20,610 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:20,610 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:20,610 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:20,610 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:20,612 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:20,612 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:20,612 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:20,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:20,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:20,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:20,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:20,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:20,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:20,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:20,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:20,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:20,613 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:20,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:20,614 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:20,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:20,616 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:20,616 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:20,616 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:20,616 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:20,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:20,616 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:20,616 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:20,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:20,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:20,616 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:20,616 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:20,617 DEBUG [Thread-2432] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:20,617 DEBUG [Thread-2432] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:20,619 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:20,616 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:20,616 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:20,620 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:20,620 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:20,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:20,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:20,660 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/A/a3bdbb5652f0492981ae9d35148ee94e as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/a3bdbb5652f0492981ae9d35148ee94e 2023-04-12 15:55:20,664 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d7d8c2ad266b62b02ebd2f4b9a678b1a/A of d7d8c2ad266b62b02ebd2f4b9a678b1a into a3bdbb5652f0492981ae9d35148ee94e(size=25.2 K), total size for store is 50.4 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-12 15:55:20,664 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d7d8c2ad266b62b02ebd2f4b9a678b1a: 2023-04-12 15:55:20,664 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a., storeName=d7d8c2ad266b62b02ebd2f4b9a678b1a/A, priority=13, startTime=1681314919413; duration=1sec 2023-04-12 15:55:20,664 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:55:20,683 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:20,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59804 deadline: 1681314980682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:20,700 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:20,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59810 deadline: 1681314980697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:20,717 DEBUG [Thread-2432] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:20,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:20,719 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:20,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:20,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:20,720 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:20,720 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:20,720 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:20,720 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:20,721 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:20,721 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:20,723 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:20,723 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:20,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:20,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:20,723 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:20,723 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:20,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:20,723 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:20,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:20,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:20,724 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:20,725 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:20,725 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:20,725 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:20,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59790 deadline: 1681314980722, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:20,725 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:20,725 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:20,725 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:20,725 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:20,725 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:20,725 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:20,725 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. started... 2023-04-12 15:55:20,725 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. as already flushing 2023-04-12 15:55:20,725 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:20,725 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:20,725 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:20,725 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:20,725 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:20,725 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:20,727 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:20,727 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:20,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:20,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:20,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:20,727 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:20,727 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:20,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:20,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:20,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:20,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:20,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:20,728 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:20,728 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72cfd8c7[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:20,728 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:20,729 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:20,729 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:20,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:20,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:20,730 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:20,730 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:20,730 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:20,730 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:20,730 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:20,731 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:20,731 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:20,731 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:20,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:20,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:20,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:20,731 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:20,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:20,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:20,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:20,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:20,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:20,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:20,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:20,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:20,734 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:20,734 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:20,734 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:20,734 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:20,734 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:20,734 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:20,737 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:20,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:20,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:20,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:20,737 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:20,737 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:20,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:20,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:20,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:20,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:20,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:20,738 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:20,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:20,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:20,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:20,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:20,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:20,740 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:20,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:20,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:20,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:20,743 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:20,743 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:20,743 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:20,743 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:20,743 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:20,743 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:20,743 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:20,743 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:20,743 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:20,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:20,743 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:20,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:20,743 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:20,743 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:20,743 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:20,744 DEBUG [Thread-2432] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:20,744 DEBUG [Thread-2432] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:20,744 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230412e2747e803b0c437692b659c1806a32cc to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412e2747e803b0c437692b659c1806a32cc 2023-04-12 15:55:20,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:20,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:20,844 DEBUG [Thread-2432] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:20,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:20,846 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:20,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:20,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:20,848 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:20,848 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:20,848 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:20,848 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:20,848 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:20,849 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:20,850 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:20,850 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:20,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:20,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:20,850 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:20,850 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:20,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:20,853 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:20,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:20,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:20,853 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:20,864 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:20,864 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:20,864 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:20,864 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:20,864 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:20,866 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:20,867 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:20,867 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:20,867 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. started... 2023-04-12 15:55:20,867 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. as already flushing 2023-04-12 15:55:20,867 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:20,867 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:20,867 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:20,867 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:20,867 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:20,867 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:20,869 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:20,869 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:20,869 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:20,869 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:20,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:20,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:20,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:20,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:20,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:20,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:20,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:20,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:20,871 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:20,871 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71947322[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:20,871 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:20,871 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:20,873 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:20,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:20,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:20,874 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:20,874 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:20,874 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:20,874 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:20,874 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:20,875 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:20,875 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:20,875 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:20,875 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:20,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:20,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:20,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:20,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:20,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:20,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:20,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:20,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:20,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:20,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:20,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:20,879 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:20,879 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:20,879 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:20,879 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:20,879 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:20,880 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:20,884 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:20,884 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:20,884 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:20,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:20,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:20,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:20,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:20,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:20,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:20,884 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:20,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:20,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:20,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:20,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:20,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:20,886 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:20,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:20,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:20,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:20,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:20,892 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:20,892 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:20,892 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:20,892 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:20,892 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:20,892 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:20,892 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:20,892 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:20,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:20,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:20,892 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:20,892 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:20,892 DEBUG [Thread-2432] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:20,892 DEBUG [Thread-2432] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:20,892 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:20,892 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:20,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:20,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:20,893 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:20,990 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:20,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59804 deadline: 1681314980984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:20,993 DEBUG [Thread-2432] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:20,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:20,995 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:21,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:21,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:21,001 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:21,001 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:21,001 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:21,001 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:21,002 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:21,002 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:21,003 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:21,003 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:21,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:21,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:21,004 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:21,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:21,004 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:21,004 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:21,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:21,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:21,004 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:21,008 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:21,008 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:21,008 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:21,008 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:21,008 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:21,008 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:21,008 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:21,008 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:21,008 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. started... 2023-04-12 15:55:21,008 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. as already flushing 2023-04-12 15:55:21,008 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:21,008 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:21,008 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:21,009 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:21,009 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:21,009 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:21,010 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:21,010 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:21,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:21,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:21,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:21,010 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:21,010 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:21,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:21,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:21,011 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:21,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59810 deadline: 1681314981006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:21,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:21,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:21,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:21,012 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:21,012 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19df8471[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:21,012 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:21,012 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:21,014 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:21,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:21,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:21,014 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:21,015 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:21,015 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:21,015 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:21,015 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:21,017 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:21,017 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:21,017 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:21,017 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:21,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:21,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:21,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:21,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:21,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:21,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:21,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:21,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:21,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:21,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:21,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:21,019 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:21,019 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:21,019 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:21,019 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:21,019 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:21,019 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:21,021 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:21,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:21,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:21,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:21,021 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:21,021 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:21,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:21,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:21,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:21,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:21,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:21,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:21,022 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:21,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:21,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:21,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:21,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:21,024 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:21,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:21,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:21,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:21,028 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:21,029 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:21,029 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:21,028 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:21,029 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:21,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:21,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:21,029 DEBUG [Thread-2432] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:21,029 DEBUG [Thread-2432] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:21,029 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:21,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:21,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:21,032 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:21,032 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:21,032 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:21,032 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:21,032 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:21,032 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:21,032 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:21,129 DEBUG [Thread-2432] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:21,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:21,131 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:21,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:21,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:21,132 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:21,132 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:21,132 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:21,133 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:21,133 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:21,133 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:21,134 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:21,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:21,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:21,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:21,135 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:21,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:21,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:21,135 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:21,135 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:21,135 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:21,135 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:21,135 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:21,135 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:21,136 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:21,136 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:21,136 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:21,136 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:21,136 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. started... 2023-04-12 15:55:21,136 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. as already flushing 2023-04-12 15:55:21,136 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:21,136 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:21,136 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:21,136 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:21,136 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:21,136 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:21,137 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:21,138 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:21,138 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:21,142 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:21,142 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:21,142 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e723b0[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:21,142 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:21,142 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:21,142 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:21,144 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:21,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:21,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:21,144 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:21,145 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:21,145 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:21,146 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:21,146 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:21,146 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:21,146 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:21,146 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:21,146 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:21,146 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:21,146 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:21,146 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:21,146 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:21,146 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:21,148 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:21,148 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:21,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:21,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:21,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:21,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:21,148 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:21,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:21,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:21,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:21,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:21,149 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:21,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:21,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:21,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:21,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:21,151 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=98, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/A/62afe8cc6dcd4d8ba4716ed8b68b9f05 2023-04-12 15:55:21,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:21,151 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:21,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:21,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:21,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:21,155 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:21,155 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:21,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:21,155 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:21,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:21,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:21,155 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:21,155 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:21,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:21,155 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:21,155 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:21,155 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:21,155 DEBUG [Thread-2432] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:21,155 DEBUG [Thread-2432] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:21,155 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:21,156 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:21,156 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:21,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:21,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:21,161 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=98 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/B/6e80215966f14aa2ad8dba34bd33a971 2023-04-12 15:55:21,170 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=98 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/C/097b8c5ea1b647adafa8dd7c6f2768ba 2023-04-12 15:55:21,178 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/A/62afe8cc6dcd4d8ba4716ed8b68b9f05 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/62afe8cc6dcd4d8ba4716ed8b68b9f05 2023-04-12 15:55:21,183 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/62afe8cc6dcd4d8ba4716ed8b68b9f05, entries=250, sequenceid=98, filesize=38.7 K 2023-04-12 15:55:21,184 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/B/6e80215966f14aa2ad8dba34bd33a971 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/6e80215966f14aa2ad8dba34bd33a971 2023-04-12 15:55:21,187 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/6e80215966f14aa2ad8dba34bd33a971, entries=150, sequenceid=98, filesize=11.6 K 2023-04-12 15:55:21,188 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/C/097b8c5ea1b647adafa8dd7c6f2768ba as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/097b8c5ea1b647adafa8dd7c6f2768ba 2023-04-12 15:55:21,191 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/097b8c5ea1b647adafa8dd7c6f2768ba, entries=150, sequenceid=98, filesize=11.6 K 2023-04-12 15:55:21,192 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for d7d8c2ad266b62b02ebd2f4b9a678b1a in 868ms, sequenceid=98, compaction requested=true 2023-04-12 15:55:21,192 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d7d8c2ad266b62b02ebd2f4b9a678b1a: 2023-04-12 15:55:21,192 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 15:55:21,192 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:55:21,193 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 15:55:21,193 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:55:21,193 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 15:55:21,194 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1912): d7d8c2ad266b62b02ebd2f4b9a678b1a/B is initiating minor compaction (all files) 2023-04-12 15:55:21,194 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d7d8c2ad266b62b02ebd2f4b9a678b1a/B in TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:21,194 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/d9f56c4fd8294b0b81536f3321e977b4, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/d31c50b168fc4069b43b9418ccdcf464, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/6e80215966f14aa2ad8dba34bd33a971] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp, totalSize=34.9 K 2023-04-12 15:55:21,193 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 91225 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 15:55:21,194 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1912): d7d8c2ad266b62b02ebd2f4b9a678b1a/A is initiating minor compaction (all files) 2023-04-12 15:55:21,194 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d7d8c2ad266b62b02ebd2f4b9a678b1a/A in TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:21,194 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/a3bdbb5652f0492981ae9d35148ee94e, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/2b512416a6da4c3c8c4aee846f2ca3d7, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/62afe8cc6dcd4d8ba4716ed8b68b9f05] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp, totalSize=89.1 K 2023-04-12 15:55:21,195 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting d9f56c4fd8294b0b81536f3321e977b4, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1681314917529 2023-04-12 15:55:21,195 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting a3bdbb5652f0492981ae9d35148ee94e, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1681314917529 2023-04-12 15:55:21,195 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:55:21,195 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting d31c50b168fc4069b43b9418ccdcf464, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=81, earliestPutTs=1681314918176 2023-04-12 15:55:21,195 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 2b512416a6da4c3c8c4aee846f2ca3d7, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=81, earliestPutTs=1681314918176 2023-04-12 15:55:21,196 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 6e80215966f14aa2ad8dba34bd33a971, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1681314919562 2023-04-12 15:55:21,196 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 62afe8cc6dcd4d8ba4716ed8b68b9f05, keycount=250, bloomtype=ROW, size=38.7 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1681314919562 2023-04-12 15:55:21,203 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] throttle.PressureAwareThroughputController(145): d7d8c2ad266b62b02ebd2f4b9a678b1a#B#compaction#335 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:55:21,250 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] throttle.PressureAwareThroughputController(145): d7d8c2ad266b62b02ebd2f4b9a678b1a#A#compaction#336 average throughput is 0.46 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:55:21,256 DEBUG [Thread-2432] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:21,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:21,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:21,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:21,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:21,259 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:21,259 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:21,259 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:21,259 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:21,259 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:21,259 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:21,262 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:21,262 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:21,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:21,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:21,262 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:21,262 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:21,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:21,262 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:21,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:21,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:21,263 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:21,263 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:21,263 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:21,263 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:21,263 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:21,263 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:21,269 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:21,269 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:21,269 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:21,270 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. started... 2023-04-12 15:55:21,270 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d7d8c2ad266b62b02ebd2f4b9a678b1a 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-12 15:55:21,270 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7d8c2ad266b62b02ebd2f4b9a678b1a, store=A 2023-04-12 15:55:21,270 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:21,270 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7d8c2ad266b62b02ebd2f4b9a678b1a, store=B 2023-04-12 15:55:21,270 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:21,270 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7d8c2ad266b62b02ebd2f4b9a678b1a, store=C 2023-04-12 15:55:21,270 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:21,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on d7d8c2ad266b62b02ebd2f4b9a678b1a 2023-04-12 15:55:21,496 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. as already flushing 2023-04-12 15:55:21,547 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:21,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59804 deadline: 1681314981541, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:21,550 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:21,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59810 deadline: 1681314981544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:21,648 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:21,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59804 deadline: 1681314981648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:21,654 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/B/a45629ce035b4ed492ff7439438f0696 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/a45629ce035b4ed492ff7439438f0696 2023-04-12 15:55:21,655 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:21,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59810 deadline: 1681314981651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:21,659 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d7d8c2ad266b62b02ebd2f4b9a678b1a/B of d7d8c2ad266b62b02ebd2f4b9a678b1a into a45629ce035b4ed492ff7439438f0696(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:55:21,660 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d7d8c2ad266b62b02ebd2f4b9a678b1a: 2023-04-12 15:55:21,660 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a., storeName=d7d8c2ad266b62b02ebd2f4b9a678b1a/B, priority=13, startTime=1681314921192; duration=0sec 2023-04-12 15:55:21,660 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:55:21,660 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 15:55:21,661 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 15:55:21,661 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1912): d7d8c2ad266b62b02ebd2f4b9a678b1a/C is initiating minor compaction (all files) 2023-04-12 15:55:21,661 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d7d8c2ad266b62b02ebd2f4b9a678b1a/C in TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:21,661 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/0de2554cf73244a8b34114410e9bd747, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/7c26a31a9e8540409e1fa547f90c6ce0, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/097b8c5ea1b647adafa8dd7c6f2768ba] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp, totalSize=34.9 K 2023-04-12 15:55:21,661 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 0de2554cf73244a8b34114410e9bd747, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1681314917529 2023-04-12 15:55:21,667 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 7c26a31a9e8540409e1fa547f90c6ce0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=81, earliestPutTs=1681314918176 2023-04-12 15:55:21,667 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 097b8c5ea1b647adafa8dd7c6f2768ba, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1681314919562 2023-04-12 15:55:21,677 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] throttle.PressureAwareThroughputController(145): d7d8c2ad266b62b02ebd2f4b9a678b1a#C#compaction#338 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:55:21,677 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/A/2fc54e85e7564141ac2214ab1e687675 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/2fc54e85e7564141ac2214ab1e687675 2023-04-12 15:55:21,683 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d7d8c2ad266b62b02ebd2f4b9a678b1a/A of d7d8c2ad266b62b02ebd2f4b9a678b1a into 2fc54e85e7564141ac2214ab1e687675(size=25.3 K), total size for store is 25.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:55:21,683 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d7d8c2ad266b62b02ebd2f4b9a678b1a: 2023-04-12 15:55:21,683 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a., storeName=d7d8c2ad266b62b02ebd2f4b9a678b1a/A, priority=13, startTime=1681314921192; duration=0sec 2023-04-12 15:55:21,684 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:55:21,688 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/C/fba63c08abfa4e8ba93b0e185a4c7a53 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/fba63c08abfa4e8ba93b0e185a4c7a53 2023-04-12 15:55:21,697 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d7d8c2ad266b62b02ebd2f4b9a678b1a/C of d7d8c2ad266b62b02ebd2f4b9a678b1a into fba63c08abfa4e8ba93b0e185a4c7a53(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:55:21,697 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d7d8c2ad266b62b02ebd2f4b9a678b1a: 2023-04-12 15:55:21,697 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a., storeName=d7d8c2ad266b62b02ebd2f4b9a678b1a/C, priority=13, startTime=1681314921195; duration=0sec 2023-04-12 15:55:21,697 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:55:21,708 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230412a57322dec6834032a01366ce18eca06f to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412a57322dec6834032a01366ce18eca06f 2023-04-12 15:55:21,734 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:21,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59790 deadline: 1681314981728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:21,856 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:21,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59804 deadline: 1681314981849, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:21,857 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:21,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59810 deadline: 1681314981857, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:22,126 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=116, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/A/51f72dfe3a574383a083a77dd4aa729f 2023-04-12 15:55:22,137 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=116 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/B/b6e3ae710afb4438b471f301094b3067 2023-04-12 15:55:22,147 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=116 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/C/622e866e408a4bea9ce2d7c712cff787 2023-04-12 15:55:22,152 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/A/51f72dfe3a574383a083a77dd4aa729f as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/51f72dfe3a574383a083a77dd4aa729f 2023-04-12 15:55:22,155 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/51f72dfe3a574383a083a77dd4aa729f, entries=150, sequenceid=116, filesize=25.1 K 2023-04-12 15:55:22,156 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/B/b6e3ae710afb4438b471f301094b3067 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/b6e3ae710afb4438b471f301094b3067 2023-04-12 15:55:22,160 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/b6e3ae710afb4438b471f301094b3067, entries=150, sequenceid=116, filesize=11.6 K 2023-04-12 15:55:22,160 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:22,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59804 deadline: 1681314982157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:22,160 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:22,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59810 deadline: 1681314982158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:22,161 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/C/622e866e408a4bea9ce2d7c712cff787 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/622e866e408a4bea9ce2d7c712cff787 2023-04-12 15:55:22,165 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/622e866e408a4bea9ce2d7c712cff787, entries=150, sequenceid=116, filesize=11.6 K 2023-04-12 15:55:22,166 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for d7d8c2ad266b62b02ebd2f4b9a678b1a in 896ms, sequenceid=116, compaction requested=false 2023-04-12 15:55:22,166 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d7d8c2ad266b62b02ebd2f4b9a678b1a: 2023-04-12 15:55:22,166 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:22,166 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:22,166 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:22,167 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:22,167 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:22,167 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:22,168 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:22,169 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:22,169 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:22,169 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:22,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:22,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:22,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:22,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:22,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:22,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:22,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:22,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:22,171 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:22,171 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@247483f7[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:22,171 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:22,171 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:22,173 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:22,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:22,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:22,173 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:22,173 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:22,174 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:22,174 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:22,174 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:22,176 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:22,176 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:22,176 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:22,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:22,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:22,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:22,176 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:22,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:22,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:22,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:22,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:22,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:22,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:22,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:22,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:22,182 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:22,182 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:22,182 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:22,182 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:22,182 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:22,182 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:22,183 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:22,184 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:22,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:22,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:22,184 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:22,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:22,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:22,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:22,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:22,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:22,184 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:22,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:22,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:22,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:22,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:22,187 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:22,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:22,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:22,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:22,189 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:22,189 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:22,189 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:22,189 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:22,189 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:22,189 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:22,189 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:22,189 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:22,189 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:22,189 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:22,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:22,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:22,189 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:22,189 DEBUG [Thread-2432] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:22,189 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:22,189 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:22,189 DEBUG [Thread-2432] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:22,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:22,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:22,290 DEBUG [Thread-2432] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:22,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:22,292 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:22,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:22,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:22,294 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:22,294 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:22,294 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:22,294 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:22,294 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:22,295 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:22,297 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:22,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:22,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:22,297 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:22,297 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:22,297 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:22,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:22,298 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:22,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:22,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:22,298 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:22,298 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:22,299 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:22,299 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:22,299 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:22,299 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:22,299 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:22,299 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:22,299 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:22,299 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. started... 2023-04-12 15:55:22,299 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d7d8c2ad266b62b02ebd2f4b9a678b1a 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-12 15:55:22,299 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7d8c2ad266b62b02ebd2f4b9a678b1a, store=A 2023-04-12 15:55:22,300 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:22,300 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7d8c2ad266b62b02ebd2f4b9a678b1a, store=B 2023-04-12 15:55:22,300 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:22,300 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7d8c2ad266b62b02ebd2f4b9a678b1a, store=C 2023-04-12 15:55:22,300 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:22,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on d7d8c2ad266b62b02ebd2f4b9a678b1a 2023-04-12 15:55:22,316 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. as already flushing 2023-04-12 15:55:22,319 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304127e5477d89335412ab1819492082c4795 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304127e5477d89335412ab1819492082c4795 2023-04-12 15:55:22,324 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=134, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/A/d26fc0e341114b6ab266042e7676a475 2023-04-12 15:55:22,332 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=134 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/B/24de0717013a46ba9a9d58a002b72cc9 2023-04-12 15:55:22,362 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:22,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59902 deadline: 1681314982360, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:22,370 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:22,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59882 deadline: 1681314982362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:22,467 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:22,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59902 deadline: 1681314982463, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:22,477 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:22,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59882 deadline: 1681314982471, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:22,665 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:22,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59810 deadline: 1681314982661, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:22,665 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:22,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59804 deadline: 1681314982662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:22,673 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:22,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59902 deadline: 1681314982669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:22,680 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:22,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59882 deadline: 1681314982678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:22,742 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=134 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/C/8a74a9c998c143e0a74e8b6a6cc3122e 2023-04-12 15:55:22,746 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/A/d26fc0e341114b6ab266042e7676a475 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/d26fc0e341114b6ab266042e7676a475 2023-04-12 15:55:22,750 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/d26fc0e341114b6ab266042e7676a475, entries=150, sequenceid=134, filesize=25.2 K 2023-04-12 15:55:22,751 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/B/24de0717013a46ba9a9d58a002b72cc9 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/24de0717013a46ba9a9d58a002b72cc9 2023-04-12 15:55:22,754 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/24de0717013a46ba9a9d58a002b72cc9, entries=150, sequenceid=134, filesize=11.7 K 2023-04-12 15:55:22,755 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/C/8a74a9c998c143e0a74e8b6a6cc3122e as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/8a74a9c998c143e0a74e8b6a6cc3122e 2023-04-12 15:55:22,758 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/8a74a9c998c143e0a74e8b6a6cc3122e, entries=150, sequenceid=134, filesize=11.7 K 2023-04-12 15:55:22,759 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for d7d8c2ad266b62b02ebd2f4b9a678b1a in 460ms, sequenceid=134, compaction requested=true 2023-04-12 15:55:22,759 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d7d8c2ad266b62b02ebd2f4b9a678b1a: 2023-04-12 15:55:22,759 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:22,759 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:22,762 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:22,762 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:22,762 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:22,762 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:22,764 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:22,764 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:22,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:22,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:22,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:22,765 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:22,765 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:22,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:22,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:22,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:22,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:22,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:22,767 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:22,767 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28716e5e[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:22,767 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:22,767 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:22,768 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:22,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:22,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:22,768 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:22,768 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:22,768 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:22,768 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:22,768 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:22,770 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:22,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:22,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:22,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:22,772 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:22,772 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:22,772 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:22,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:22,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:22,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:22,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:22,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:22,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:22,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:22,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:22,776 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:22,776 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:22,776 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:22,776 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:22,776 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:22,776 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:22,780 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:22,780 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:22,780 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:22,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:22,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:22,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:22,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:22,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:22,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:22,781 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:22,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:22,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:22,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:22,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:22,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:22,783 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:22,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:22,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:22,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:22,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:22,786 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:22,786 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:22,786 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:22,787 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:22,787 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:22,787 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:22,787 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:22,787 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:22,787 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:22,787 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:22,787 DEBUG [Thread-2432] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:22,787 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:22,787 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:22,787 DEBUG [Thread-2432] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:22,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:22,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:22,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:22,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:22,788 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:22,887 DEBUG [Thread-2432] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:22,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:22,889 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:22,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:22,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:22,890 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:22,890 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:22,890 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:22,890 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:22,891 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:22,891 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:22,892 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:22,892 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:22,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:22,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:22,892 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:22,892 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:22,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:22,893 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:22,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:22,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:22,893 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:22,893 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:22,893 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:22,894 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:22,894 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:22,894 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:22,894 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:22,894 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:22,894 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:22,894 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. started... 2023-04-12 15:55:22,894 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d7d8c2ad266b62b02ebd2f4b9a678b1a 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-12 15:55:22,894 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7d8c2ad266b62b02ebd2f4b9a678b1a, store=A 2023-04-12 15:55:22,894 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:22,895 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7d8c2ad266b62b02ebd2f4b9a678b1a, store=B 2023-04-12 15:55:22,895 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:22,895 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7d8c2ad266b62b02ebd2f4b9a678b1a, store=C 2023-04-12 15:55:22,895 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:22,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on d7d8c2ad266b62b02ebd2f4b9a678b1a 2023-04-12 15:55:22,978 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. as already flushing 2023-04-12 15:55:23,017 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:23,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59882 deadline: 1681314983008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:23,018 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:23,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59902 deadline: 1681314983009, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:23,122 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:23,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59902 deadline: 1681314983119, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:23,123 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:23,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59882 deadline: 1681314983119, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:23,317 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230412fee0eb20a6e44fae9a5d9ddd6627b0e8 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412fee0eb20a6e44fae9a5d9ddd6627b0e8 2023-04-12 15:55:23,327 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=151, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/A/d3e5c62918d04cada2d1a42c03a26711 2023-04-12 15:55:23,331 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:23,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59882 deadline: 1681314983326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:23,331 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:23,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59902 deadline: 1681314983326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:23,635 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:23,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59882 deadline: 1681314983633, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:23,635 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:23,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59902 deadline: 1681314983633, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:23,677 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:23,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59804 deadline: 1681314983671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:23,678 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:23,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59810 deadline: 1681314983675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:23,740 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=151 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/B/2312d85197b64cae80b124896647d8f0 2023-04-12 15:55:23,750 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:23,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59790 deadline: 1681314983747, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:23,751 DEBUG [Thread-2422] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4148 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a., hostname=jenkins-hbase4.apache.org,38905,1681314818739, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-12 15:55:24,143 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:24,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59902 deadline: 1681314984138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:24,143 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:24,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59882 deadline: 1681314984139, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:24,150 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=151 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/C/b49c1225504e4e6498581338ff9dc928 2023-04-12 15:55:24,155 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/A/d3e5c62918d04cada2d1a42c03a26711 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/d3e5c62918d04cada2d1a42c03a26711 2023-04-12 15:55:24,160 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/d3e5c62918d04cada2d1a42c03a26711, entries=150, sequenceid=151, filesize=25.3 K 2023-04-12 15:55:24,162 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/B/2312d85197b64cae80b124896647d8f0 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/2312d85197b64cae80b124896647d8f0 2023-04-12 15:55:24,166 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/2312d85197b64cae80b124896647d8f0, entries=150, sequenceid=151, filesize=11.8 K 2023-04-12 15:55:24,167 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/C/b49c1225504e4e6498581338ff9dc928 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/b49c1225504e4e6498581338ff9dc928 2023-04-12 15:55:24,177 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/b49c1225504e4e6498581338ff9dc928, entries=150, sequenceid=151, filesize=11.8 K 2023-04-12 15:55:24,177 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for d7d8c2ad266b62b02ebd2f4b9a678b1a in 1283ms, sequenceid=151, compaction requested=true 2023-04-12 15:55:24,177 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d7d8c2ad266b62b02ebd2f4b9a678b1a: 2023-04-12 15:55:24,177 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:24,177 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:24,177 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:24,177 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:24,178 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:24,178 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:24,179 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:24,179 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:24,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:24,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:24,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:24,179 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:24,179 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:24,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:24,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:24,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:24,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:24,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:24,182 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:24,183 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7aae0511[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:24,183 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:24,183 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:24,184 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:24,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:24,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:24,184 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:24,184 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:24,184 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:24,184 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:24,184 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:24,185 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:24,185 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:24,185 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:24,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:24,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:24,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:24,186 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:24,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:24,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:24,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:24,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:24,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:24,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:24,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:24,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:24,189 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:24,189 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:24,189 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:24,189 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:24,189 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:24,189 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:24,192 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:24,192 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:24,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:24,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:24,192 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:24,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:24,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:24,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:24,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:24,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:24,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:24,193 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:24,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:24,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:24,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:24,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:24,195 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:24,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:24,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:24,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:24,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:24,198 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:24,198 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:24,198 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:24,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:24,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:24,198 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:24,198 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:24,198 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:24,198 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:24,198 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:24,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:24,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:24,198 DEBUG [Thread-2432] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:24,198 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:24,198 DEBUG [Thread-2432] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:24,198 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:24,198 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:24,198 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:24,198 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:24,298 DEBUG [Thread-2432] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:24,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:24,301 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:24,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:24,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:24,302 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:24,302 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:24,302 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:24,302 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:24,302 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:24,302 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:24,303 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:24,303 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:24,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:24,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:24,304 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:24,304 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:24,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:24,304 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:24,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:24,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:24,304 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:24,304 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:24,304 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:24,305 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:24,305 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:24,305 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:24,305 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:24,305 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:24,305 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:24,305 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. started... 2023-04-12 15:55:24,305 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d7d8c2ad266b62b02ebd2f4b9a678b1a 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-12 15:55:24,305 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7d8c2ad266b62b02ebd2f4b9a678b1a, store=A 2023-04-12 15:55:24,305 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:24,305 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7d8c2ad266b62b02ebd2f4b9a678b1a, store=B 2023-04-12 15:55:24,305 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:24,306 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7d8c2ad266b62b02ebd2f4b9a678b1a, store=C 2023-04-12 15:55:24,306 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:24,328 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041268155fef94834d0ea6ad4e6480b39806 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041268155fef94834d0ea6ad4e6480b39806 2023-04-12 15:55:24,734 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=166, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/A/4a04250de0db44d7bf907caeebc7d05c 2023-04-12 15:55:25,148 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=166 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/B/32ad9c3712c24119ac304e665319a379 2023-04-12 15:55:25,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on d7d8c2ad266b62b02ebd2f4b9a678b1a 2023-04-12 15:55:25,157 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. as already flushing 2023-04-12 15:55:25,203 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:25,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59902 deadline: 1681314985198, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:25,205 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:25,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59882 deadline: 1681314985199, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:25,311 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:25,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59902 deadline: 1681314985304, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:25,311 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:25,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59882 deadline: 1681314985306, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:25,517 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:25,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59902 deadline: 1681314985512, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:25,517 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:25,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59882 deadline: 1681314985513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:25,561 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=166 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/C/bd1fe030bee8418d953f2c3742630f7f 2023-04-12 15:55:25,565 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/A/4a04250de0db44d7bf907caeebc7d05c as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/4a04250de0db44d7bf907caeebc7d05c 2023-04-12 15:55:25,568 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/4a04250de0db44d7bf907caeebc7d05c, entries=150, sequenceid=166, filesize=25.3 K 2023-04-12 15:55:25,569 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/B/32ad9c3712c24119ac304e665319a379 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/32ad9c3712c24119ac304e665319a379 2023-04-12 15:55:25,573 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/32ad9c3712c24119ac304e665319a379, entries=150, sequenceid=166, filesize=11.8 K 2023-04-12 15:55:25,574 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/C/bd1fe030bee8418d953f2c3742630f7f as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/bd1fe030bee8418d953f2c3742630f7f 2023-04-12 15:55:25,577 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/bd1fe030bee8418d953f2c3742630f7f, entries=150, sequenceid=166, filesize=11.8 K 2023-04-12 15:55:25,578 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for d7d8c2ad266b62b02ebd2f4b9a678b1a in 1273ms, sequenceid=166, compaction requested=true 2023-04-12 15:55:25,578 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d7d8c2ad266b62b02ebd2f4b9a678b1a: 2023-04-12 15:55:25,578 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:25,578 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:25,578 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:25,578 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:25,578 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:25,578 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:25,580 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:25,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:25,580 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:25,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:25,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:25,580 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:25,580 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:25,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:25,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:25,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:25,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:25,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:25,582 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:25,582 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1fb7ea39[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:25,582 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:25,582 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:25,583 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:25,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:25,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:25,583 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:25,583 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:25,583 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:25,583 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:25,583 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:25,584 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:25,584 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:25,584 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:25,584 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:25,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:25,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:25,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:25,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:25,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:25,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:25,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:25,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:25,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:25,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:25,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:25,587 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:25,587 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:25,587 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:25,587 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:25,587 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:25,587 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:25,589 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:25,589 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:25,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:25,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:25,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:25,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:25,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:25,589 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:25,590 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:25,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:25,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:25,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:25,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:25,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:25,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:25,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:25,591 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:25,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:25,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:25,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:25,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:25,594 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:25,594 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:25,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:25,594 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:25,594 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:25,594 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:25,594 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:25,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:25,594 DEBUG [Thread-2432] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:25,594 DEBUG [Thread-2432] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:25,594 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:25,594 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:25,594 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:25,594 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:25,594 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:25,594 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:25,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:25,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:25,694 DEBUG [Thread-2432] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:25,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:25,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on d7d8c2ad266b62b02ebd2f4b9a678b1a 2023-04-12 15:55:25,695 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d7d8c2ad266b62b02ebd2f4b9a678b1a 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-12 15:55:25,696 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7d8c2ad266b62b02ebd2f4b9a678b1a, store=A 2023-04-12 15:55:25,696 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:25,696 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7d8c2ad266b62b02ebd2f4b9a678b1a, store=B 2023-04-12 15:55:25,696 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:25,696 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7d8c2ad266b62b02ebd2f4b9a678b1a, store=C 2023-04-12 15:55:25,696 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:25,696 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:25,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:25,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:25,698 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:25,698 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:25,698 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:25,698 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:25,698 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:25,698 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:25,702 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:25,704 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:25,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:25,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:25,712 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:25,712 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:25,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:25,712 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:25,713 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230412a08092d048a6471daa35686cf6b36315 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412a08092d048a6471daa35686cf6b36315 2023-04-12 15:55:25,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:25,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:25,713 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:25,713 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:25,713 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:25,713 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:25,713 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:25,713 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:25,714 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:25,714 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:25,714 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:25,714 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. started... 2023-04-12 15:55:25,714 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. as already flushing 2023-04-12 15:55:25,714 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:25,714 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:25,714 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:25,714 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:25,714 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:25,714 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:25,716 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:25,716 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:25,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:25,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:25,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:25,716 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:25,716 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:25,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:25,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:25,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:25,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:25,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:25,718 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:25,718 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f5cb903[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:25,718 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:25,718 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:25,719 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:25,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:25,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:25,719 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:25,719 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:25,719 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:25,719 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:25,719 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:25,720 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:25,720 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:25,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:25,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:25,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:25,720 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:25,721 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:25,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:25,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:25,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:25,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:25,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:25,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:25,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:25,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:25,722 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:25,722 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:25,723 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:25,723 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:25,723 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:25,723 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:25,725 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:25,726 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:25,728 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:25,728 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:25,728 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:25,729 DEBUG [Thread-2432] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:25,729 DEBUG [Thread-2432] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:25,732 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:25,732 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:25,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:25,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:25,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:25,732 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:25,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:25,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:25,732 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:25,732 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:25,732 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:25,732 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:25,732 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:25,732 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:25,732 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:25,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:25,732 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:25,733 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:25,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:25,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:25,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:25,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:25,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:25,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:25,733 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:25,743 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:25,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59804 deadline: 1681314985736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:25,744 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:25,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59810 deadline: 1681314985740, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:25,824 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:25,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59882 deadline: 1681314985818, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:25,825 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:25,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59902 deadline: 1681314985820, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:25,829 DEBUG [Thread-2432] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:25,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:25,831 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:25,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:25,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:25,840 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:25,840 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:25,840 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:25,840 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:25,840 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:25,840 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:25,842 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:25,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:25,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:25,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:25,842 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:25,843 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:25,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:25,843 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:25,843 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:25,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:25,843 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:25,843 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:25,843 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:25,843 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:25,843 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:25,843 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:25,844 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:25,844 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:25,844 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:25,844 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. started... 2023-04-12 15:55:25,845 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. as already flushing 2023-04-12 15:55:25,845 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:25,845 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:25,845 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:25,845 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:25,845 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:25,845 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:25,846 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:25,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:25,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:25,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:25,846 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:25,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:25,847 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:25,847 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:25,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:25,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:25,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:25,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:25,852 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:25,852 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3cfc3728[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:25,852 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:25,852 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:25,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59804 deadline: 1681314985845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:25,852 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:25,852 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:25,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59810 deadline: 1681314985845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:25,853 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:25,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:25,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:25,853 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:25,853 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:25,853 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:25,853 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:25,854 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:25,856 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:25,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:25,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:25,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:25,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:25,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:25,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:25,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:25,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:25,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:25,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:25,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:25,859 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:25,859 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:25,859 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:25,859 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:25,859 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:25,859 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:25,860 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:25,860 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:25,860 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:25,861 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:25,861 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:25,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:25,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:25,861 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:25,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:25,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:25,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:25,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:25,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:25,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:25,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:25,862 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:25,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:25,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:25,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:25,863 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:25,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:25,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:25,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:25,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:25,867 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:25,867 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:25,868 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:25,868 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:25,868 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:25,868 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:25,868 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:25,868 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:25,868 DEBUG [Thread-2432] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:25,868 DEBUG [Thread-2432] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:25,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:25,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:25,868 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:25,869 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:25,869 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:25,869 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:25,869 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:25,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:25,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:25,968 DEBUG [Thread-2432] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:25,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:25,971 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:25,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:25,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:25,980 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:25,980 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:25,980 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:25,980 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:25,980 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:25,981 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:25,982 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:25,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:25,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:25,982 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:25,983 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:25,983 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:25,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:25,983 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:25,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:25,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:25,984 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:25,988 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:25,988 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:25,988 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:25,988 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:25,988 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:25,991 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:25,991 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:25,991 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:25,991 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. started... 2023-04-12 15:55:25,991 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. as already flushing 2023-04-12 15:55:25,991 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:25,991 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:25,991 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:25,992 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:25,992 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:25,992 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:25,994 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:25,994 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:25,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:25,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:25,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:25,994 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:25,994 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:25,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:25,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:25,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:25,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:25,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:25,995 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:25,995 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36f03b2b[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:25,995 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:25,995 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:25,996 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:25,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:25,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:25,997 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:25,997 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:25,997 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:25,997 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:25,997 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:25,998 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:25,998 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:25,998 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:25,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:25,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:25,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:25,998 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:25,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:25,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:25,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:25,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:25,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:25,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:26,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:26,000 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:26,000 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:26,000 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:26,000 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:26,000 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:26,000 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:26,002 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:26,002 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:26,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:26,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:26,002 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:26,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:26,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:26,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:26,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:26,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:26,002 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:26,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:26,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:26,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:26,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:26,003 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:26,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:26,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,007 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,007 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:26,007 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:26,007 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:26,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:26,007 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:26,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:26,007 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:26,007 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,008 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:26,008 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:26,008 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:26,007 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:26,007 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:26,008 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:26,008 DEBUG [Thread-2432] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:26,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:26,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:26,008 DEBUG [Thread-2432] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:26,057 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:26,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59804 deadline: 1681314986053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,057 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:26,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59810 deadline: 1681314986053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,108 DEBUG [Thread-2432] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:26,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:26,112 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:26,128 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=184, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/A/2406a5d42e7744ef8a79db3374b20945 2023-04-12 15:55:26,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:26,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:26,133 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:26,133 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:26,133 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:26,133 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:26,135 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:26,135 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:26,137 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:26,137 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:26,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:26,137 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,137 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:26,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:26,138 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:26,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:26,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:26,138 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:26,148 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:26,148 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:26,148 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:26,148 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:26,148 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:26,148 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:26,148 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:26,149 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:26,149 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. started... 2023-04-12 15:55:26,150 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. as already flushing 2023-04-12 15:55:26,150 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:26,150 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:26,150 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:26,150 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,153 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:26,153 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:26,160 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:26,160 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:26,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:26,161 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:26,161 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:26,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:26,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:26,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:26,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:26,163 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:26,163 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16b6e9ee[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:26,163 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:26,164 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:26,166 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:26,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:26,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:26,166 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:26,166 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:26,166 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:26,166 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,166 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:26,167 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:26,167 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,167 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:26,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:26,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:26,167 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:26,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:26,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:26,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:26,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:26,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:26,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:26,171 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:26,171 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:26,171 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:26,171 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:26,171 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:26,171 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:26,173 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:26,173 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:26,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:26,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:26,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:26,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:26,174 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:26,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:26,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:26,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:26,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:26,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:26,176 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:26,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:26,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:26,179 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:26,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:26,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,184 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,184 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:26,184 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:26,184 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:26,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:26,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:26,184 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:26,184 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:26,184 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:26,184 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:26,184 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,185 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:26,185 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:26,185 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:26,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:26,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:26,185 DEBUG [Thread-2432] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:26,185 DEBUG [Thread-2432] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:26,185 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:26,285 DEBUG [Thread-2432] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:26,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:26,287 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:26,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:26,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:26,292 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:26,292 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:26,292 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:26,292 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:26,292 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:26,293 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:26,295 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:26,295 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:26,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:26,296 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,296 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:26,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:26,296 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:26,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:26,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:26,297 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:26,300 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:26,300 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:26,300 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:26,300 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:26,300 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:26,304 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:26,304 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:26,304 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:26,304 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. started... 2023-04-12 15:55:26,304 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. as already flushing 2023-04-12 15:55:26,304 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:26,304 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:26,304 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:26,304 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,304 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:26,305 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:26,306 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,306 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:26,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:26,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:26,306 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:26,306 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:26,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:26,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:26,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:26,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:26,308 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:26,308 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c58de9a[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:26,308 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:26,308 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:26,309 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:26,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:26,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:26,309 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:26,309 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:26,309 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:26,309 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,309 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:26,310 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,310 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:26,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:26,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:26,311 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:26,311 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:26,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:26,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:26,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:26,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:26,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:26,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:26,314 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:26,314 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:26,314 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:26,314 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:26,314 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:26,314 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:26,317 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:26,317 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:26,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:26,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:26,317 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:26,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:26,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:26,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:26,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:26,317 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:26,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:26,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:26,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:26,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:26,319 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:26,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:26,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:26,320 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,320 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:26,320 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:26,321 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:26,321 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:26,321 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:26,321 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:26,321 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,321 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:26,321 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:26,321 DEBUG [Thread-2432] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:26,321 DEBUG [Thread-2432] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:26,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:26,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:26,321 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:26,321 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:26,321 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:26,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:26,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:26,331 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:26,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59882 deadline: 1681314986325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,337 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:26,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59902 deadline: 1681314986328, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,367 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:26,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59804 deadline: 1681314986359, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,367 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:26,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59810 deadline: 1681314986360, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,421 DEBUG [Thread-2432] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:26,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:26,423 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:26,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:26,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:26,424 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:26,425 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:26,425 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:26,425 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:26,425 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:26,425 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:26,426 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:26,426 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:26,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:26,427 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,427 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:26,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:26,427 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:26,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:26,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:26,428 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:26,428 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:26,428 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:26,428 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:26,428 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:26,428 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:26,428 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:26,428 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:26,429 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:26,429 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. started... 2023-04-12 15:55:26,429 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. as already flushing 2023-04-12 15:55:26,429 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:26,429 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:26,429 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:26,429 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,429 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:26,429 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:26,430 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,430 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:26,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:26,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:26,431 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:26,431 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:26,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:26,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:26,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:26,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:26,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:26,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31c0e14e[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:26,433 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:26,433 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:26,435 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:26,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:26,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:26,436 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:26,436 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:26,436 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:26,436 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,436 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:26,437 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:26,438 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:26,438 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:26,438 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:26,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:26,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:26,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:26,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:26,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:26,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:26,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:26,441 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:26,441 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:26,441 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:26,441 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:26,441 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:26,441 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:26,442 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:26,442 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:26,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:26,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:26,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:26,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:26,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:26,443 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,446 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,446 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:26,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:26,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:26,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:26,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:26,448 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,448 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:26,448 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:26,448 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:26,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:26,448 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,448 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:26,448 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:26,448 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:26,448 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:26,448 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:26,448 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:26,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:26,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:26,448 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:26,449 DEBUG [Thread-2432] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:26,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:26,449 DEBUG [Thread-2432] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:26,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:26,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:26,449 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:26,549 DEBUG [Thread-2432] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:26,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:26,551 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:26,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:26,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:26,552 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:26,552 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:26,552 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:26,552 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:26,552 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:26,552 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:26,554 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:26,554 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:26,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:26,555 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,555 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:26,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:26,555 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:26,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:26,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:26,556 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:26,556 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=184 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/B/eed82f35804f44ce8fe1a24eba3a28a9 2023-04-12 15:55:26,556 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:26,556 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:26,556 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:26,556 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:26,556 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:26,556 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:26,557 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:26,557 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:26,557 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. started... 2023-04-12 15:55:26,557 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. as already flushing 2023-04-12 15:55:26,557 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:26,557 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:26,557 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:26,557 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,557 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:26,557 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:26,559 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,559 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:26,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:26,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:26,559 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:26,559 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:26,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:26,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:26,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:26,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:26,561 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:26,561 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6debd3e[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:26,561 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:26,561 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:26,562 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:26,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:26,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:26,562 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:26,562 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:26,562 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:26,562 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,562 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:26,570 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:26,570 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:26,570 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,570 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:26,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:26,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:26,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:26,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:26,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:26,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:26,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:26,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:26,572 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:26,572 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:26,572 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:26,572 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:26,572 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:26,572 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:26,574 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:26,574 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:26,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:26,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:26,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:26,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:26,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:26,574 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:26,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:26,574 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:26,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:26,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:26,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:26,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:26,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:26,575 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:26,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:26,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,577 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:26,577 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:26,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:26,577 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:26,577 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:26,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:26,577 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,578 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:26,578 DEBUG [Thread-2432] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:26,578 DEBUG [Thread-2432] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:26,577 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:26,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:26,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:26,578 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:26,578 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:26,578 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,578 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:26,578 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:26,578 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:26,678 DEBUG [Thread-2432] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:26,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:26,681 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:26,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:26,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:26,682 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:26,682 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:26,682 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:26,682 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:26,682 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:26,682 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:26,684 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,684 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:26,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:26,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:26,684 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,684 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:26,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:26,685 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:26,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:26,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:26,686 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:26,686 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:26,686 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:26,686 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:26,686 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:26,686 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:26,686 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:26,686 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:26,686 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:26,687 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. started... 2023-04-12 15:55:26,687 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. as already flushing 2023-04-12 15:55:26,687 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:26,687 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:26,687 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:26,687 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,687 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:26,687 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:26,689 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,689 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:26,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:26,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:26,690 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:26,690 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:26,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:26,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:26,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:26,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:26,692 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:26,692 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2786cd92[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:26,692 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:26,692 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:26,693 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:26,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:26,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:26,693 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:26,693 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:26,693 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:26,693 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,693 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:26,695 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:26,695 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:26,695 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:26,695 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:26,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:26,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:26,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:26,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:26,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:26,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:26,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:26,699 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:26,699 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:26,699 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:26,699 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:26,699 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:26,699 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:26,701 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:26,701 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:26,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:26,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:26,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:26,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:26,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:26,701 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:26,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:26,702 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:26,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:26,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:26,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:26,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:26,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:26,703 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:26,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:26,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,707 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,707 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:26,707 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:26,707 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:26,708 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:26,708 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:26,707 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:26,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:26,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:26,707 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:26,708 DEBUG [Thread-2432] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:26,708 DEBUG [Thread-2432] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:26,708 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,708 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:26,708 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:26,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:26,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:26,708 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:26,808 DEBUG [Thread-2432] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:26,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:26,810 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:26,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:26,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:26,811 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:26,811 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:26,811 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:26,811 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:26,812 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:26,812 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:26,813 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,813 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:26,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:26,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:26,813 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,813 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:26,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:26,814 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:26,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:26,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:26,814 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:26,814 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:26,814 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:26,814 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:26,814 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:26,814 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:26,815 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:26,815 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:26,815 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:26,815 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. started... 2023-04-12 15:55:26,815 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. as already flushing 2023-04-12 15:55:26,815 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:26,815 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:26,815 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:26,815 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,815 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:26,815 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:26,816 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,816 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:26,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:26,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:26,817 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:26,817 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:26,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:26,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:26,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:26,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:26,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:26,818 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:26,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d806bf6[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:26,818 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:26,819 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:26,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:26,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:26,819 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:26,819 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:26,819 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:26,819 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,820 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:26,826 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,826 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:26,826 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:26,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:26,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:26,826 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:26,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:26,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:26,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:26,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:26,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:26,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:26,829 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:26,829 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:26,829 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:26,829 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:26,829 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:26,829 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:26,830 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:26,830 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:26,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:26,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:26,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:26,830 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:26,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:26,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:26,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:26,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:26,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:26,831 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:26,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:26,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:26,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:26,832 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:26,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:26,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,834 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:26,834 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:26,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:26,835 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:26,835 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:26,835 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,835 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:26,835 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:26,835 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,835 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:26,835 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:26,835 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:26,835 DEBUG [Thread-2432] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:26,835 DEBUG [Thread-2432] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:26,835 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:26,835 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:26,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:26,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:26,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:26,877 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:26,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59804 deadline: 1681314986871, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,877 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:26,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59810 deadline: 1681314986872, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,935 DEBUG [Thread-2432] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:26,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:26,937 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:26,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:26,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:26,940 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:26,940 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:26,940 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:26,940 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:26,940 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:26,940 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:26,941 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:26,941 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:26,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:26,942 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,942 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:26,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:26,942 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:26,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:26,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:26,943 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:26,951 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:26,951 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:26,952 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:26,952 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:26,952 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:26,952 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:26,952 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:26,952 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:26,952 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. started... 2023-04-12 15:55:26,952 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. as already flushing 2023-04-12 15:55:26,952 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:26,952 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:26,952 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:26,952 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,952 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:26,952 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:26,954 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,954 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:26,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:26,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:26,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:26,955 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:26,955 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:26,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:26,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:26,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:26,956 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:26,956 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3bfc2199[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:26,956 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:26,956 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:26,958 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:26,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:26,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:26,958 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:26,958 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:26,958 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:26,958 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,958 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:26,959 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,959 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:26,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:26,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:26,959 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:26,960 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:26,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:26,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:26,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:26,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:26,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:26,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:26,962 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:26,962 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:26,962 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:26,962 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:26,962 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:26,962 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:26,967 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:26,967 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:26,967 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:26,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:26,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:26,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:26,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:26,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:26,968 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:26,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:26,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:26,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:26,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:26,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:26,969 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:26,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:26,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:26,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,970 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=184 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/C/193370cac8164b4498412057e1632419 2023-04-12 15:55:26,971 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:26,971 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:26,971 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:26,971 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:26,971 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:26,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:26,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:26,971 DEBUG [Thread-2432] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:26,971 DEBUG [Thread-2432] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:26,972 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:26,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:26,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:26,973 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,973 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:26,973 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:26,973 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:26,973 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:26,973 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:26,973 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:26,977 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/A/2406a5d42e7744ef8a79db3374b20945 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/2406a5d42e7744ef8a79db3374b20945 2023-04-12 15:55:26,980 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/2406a5d42e7744ef8a79db3374b20945, entries=200, sequenceid=184, filesize=32.1 K 2023-04-12 15:55:26,981 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/B/eed82f35804f44ce8fe1a24eba3a28a9 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/eed82f35804f44ce8fe1a24eba3a28a9 2023-04-12 15:55:26,992 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/eed82f35804f44ce8fe1a24eba3a28a9, entries=150, sequenceid=184, filesize=11.8 K 2023-04-12 15:55:26,996 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/C/193370cac8164b4498412057e1632419 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/193370cac8164b4498412057e1632419 2023-04-12 15:55:27,004 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/193370cac8164b4498412057e1632419, entries=150, sequenceid=184, filesize=11.8 K 2023-04-12 15:55:27,004 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for d7d8c2ad266b62b02ebd2f4b9a678b1a in 1309ms, sequenceid=184, compaction requested=true 2023-04-12 15:55:27,004 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d7d8c2ad266b62b02ebd2f4b9a678b1a: 2023-04-12 15:55:27,004 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:55:27,004 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-12 15:55:27,005 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-12 15:55:27,004 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:55:27,005 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:55:27,007 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 72072 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-12 15:55:27,007 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 162222 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-12 15:55:27,007 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1912): d7d8c2ad266b62b02ebd2f4b9a678b1a/B is initiating minor compaction (all files) 2023-04-12 15:55:27,008 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1912): d7d8c2ad266b62b02ebd2f4b9a678b1a/A is initiating minor compaction (all files) 2023-04-12 15:55:27,008 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d7d8c2ad266b62b02ebd2f4b9a678b1a/B in TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:27,008 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d7d8c2ad266b62b02ebd2f4b9a678b1a/A in TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:27,008 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/a45629ce035b4ed492ff7439438f0696, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/b6e3ae710afb4438b471f301094b3067, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/24de0717013a46ba9a9d58a002b72cc9, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/2312d85197b64cae80b124896647d8f0, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/32ad9c3712c24119ac304e665319a379, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/eed82f35804f44ce8fe1a24eba3a28a9] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp, totalSize=70.4 K 2023-04-12 15:55:27,008 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/2fc54e85e7564141ac2214ab1e687675, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/51f72dfe3a574383a083a77dd4aa729f, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/d26fc0e341114b6ab266042e7676a475, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/d3e5c62918d04cada2d1a42c03a26711, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/4a04250de0db44d7bf907caeebc7d05c, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/2406a5d42e7744ef8a79db3374b20945] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp, totalSize=158.4 K 2023-04-12 15:55:27,009 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting a45629ce035b4ed492ff7439438f0696, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1681314919562 2023-04-12 15:55:27,009 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 2fc54e85e7564141ac2214ab1e687675, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1681314919562 2023-04-12 15:55:27,009 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting b6e3ae710afb4438b471f301094b3067, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=116, earliestPutTs=1681314920340 2023-04-12 15:55:27,009 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 51f72dfe3a574383a083a77dd4aa729f, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=116, earliestPutTs=1681314920340 2023-04-12 15:55:27,009 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 24de0717013a46ba9a9d58a002b72cc9, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=134, earliestPutTs=1681314921517 2023-04-12 15:55:27,010 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 2312d85197b64cae80b124896647d8f0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=151, earliestPutTs=1681314922339 2023-04-12 15:55:27,010 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting d26fc0e341114b6ab266042e7676a475, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=134, earliestPutTs=1681314921517 2023-04-12 15:55:27,010 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 32ad9c3712c24119ac304e665319a379, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=166, earliestPutTs=1681314923003 2023-04-12 15:55:27,010 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting d3e5c62918d04cada2d1a42c03a26711, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=151, earliestPutTs=1681314922339 2023-04-12 15:55:27,011 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting eed82f35804f44ce8fe1a24eba3a28a9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=184, earliestPutTs=1681314925176 2023-04-12 15:55:27,011 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 4a04250de0db44d7bf907caeebc7d05c, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=166, earliestPutTs=1681314923003 2023-04-12 15:55:27,012 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 2406a5d42e7744ef8a79db3374b20945, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=184, earliestPutTs=1681314925176 2023-04-12 15:55:27,030 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] throttle.PressureAwareThroughputController(145): d7d8c2ad266b62b02ebd2f4b9a678b1a#B#compaction#353 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:55:27,039 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] throttle.PressureAwareThroughputController(145): d7d8c2ad266b62b02ebd2f4b9a678b1a#A#compaction#354 average throughput is 1.97 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:55:27,072 DEBUG [Thread-2432] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:27,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:27,074 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:27,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:27,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:27,076 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:27,076 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:27,076 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:27,076 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:27,077 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:27,077 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:27,078 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:27,078 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:27,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:27,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:27,079 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:27,079 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:27,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:27,079 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:27,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:27,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:27,079 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:27,084 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:27,084 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:27,084 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:27,084 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:27,084 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:27,088 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:27,088 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:27,088 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:27,088 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. started... 2023-04-12 15:55:27,088 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d7d8c2ad266b62b02ebd2f4b9a678b1a 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-12 15:55:27,089 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7d8c2ad266b62b02ebd2f4b9a678b1a, store=A 2023-04-12 15:55:27,089 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:27,089 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7d8c2ad266b62b02ebd2f4b9a678b1a, store=B 2023-04-12 15:55:27,089 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:27,089 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7d8c2ad266b62b02ebd2f4b9a678b1a, store=C 2023-04-12 15:55:27,089 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:27,105 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304129a34e3436bf247d4a5840ee58d54f519 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304129a34e3436bf247d4a5840ee58d54f519 2023-04-12 15:55:27,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on d7d8c2ad266b62b02ebd2f4b9a678b1a 2023-04-12 15:55:27,348 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. as already flushing 2023-04-12 15:55:27,424 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:27,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59902 deadline: 1681314987411, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:27,436 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:27,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59882 deadline: 1681314987424, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:27,445 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/B/76b6b15d792c4cd9aca15b9dd77c1336 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/76b6b15d792c4cd9aca15b9dd77c1336 2023-04-12 15:55:27,453 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in d7d8c2ad266b62b02ebd2f4b9a678b1a/B of d7d8c2ad266b62b02ebd2f4b9a678b1a into 76b6b15d792c4cd9aca15b9dd77c1336(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:55:27,453 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d7d8c2ad266b62b02ebd2f4b9a678b1a: 2023-04-12 15:55:27,453 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a., storeName=d7d8c2ad266b62b02ebd2f4b9a678b1a/B, priority=10, startTime=1681314927004; duration=0sec 2023-04-12 15:55:27,454 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:55:27,454 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-12 15:55:27,456 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 72072 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-12 15:55:27,456 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1912): d7d8c2ad266b62b02ebd2f4b9a678b1a/C is initiating minor compaction (all files) 2023-04-12 15:55:27,457 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d7d8c2ad266b62b02ebd2f4b9a678b1a/C in TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:27,457 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/fba63c08abfa4e8ba93b0e185a4c7a53, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/622e866e408a4bea9ce2d7c712cff787, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/8a74a9c998c143e0a74e8b6a6cc3122e, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/b49c1225504e4e6498581338ff9dc928, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/bd1fe030bee8418d953f2c3742630f7f, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/193370cac8164b4498412057e1632419] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp, totalSize=70.4 K 2023-04-12 15:55:27,457 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting fba63c08abfa4e8ba93b0e185a4c7a53, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1681314919562 2023-04-12 15:55:27,457 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 622e866e408a4bea9ce2d7c712cff787, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=116, earliestPutTs=1681314920340 2023-04-12 15:55:27,458 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 8a74a9c998c143e0a74e8b6a6cc3122e, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=134, earliestPutTs=1681314921517 2023-04-12 15:55:27,458 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting b49c1225504e4e6498581338ff9dc928, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=151, earliestPutTs=1681314922339 2023-04-12 15:55:27,458 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting bd1fe030bee8418d953f2c3742630f7f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=166, earliestPutTs=1681314923003 2023-04-12 15:55:27,458 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 193370cac8164b4498412057e1632419, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=184, earliestPutTs=1681314925176 2023-04-12 15:55:27,472 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] throttle.PressureAwareThroughputController(145): d7d8c2ad266b62b02ebd2f4b9a678b1a#C#compaction#356 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:55:27,517 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=198, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/A/6a67a5c9dc714fb8bae990f7bc098db7 2023-04-12 15:55:27,532 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:27,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59902 deadline: 1681314987525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:27,542 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:27,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59882 deadline: 1681314987537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:27,735 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:27,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59902 deadline: 1681314987733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:27,749 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:27,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59882 deadline: 1681314987744, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:27,765 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:27,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59790 deadline: 1681314987760, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:27,767 DEBUG [Thread-2422] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8164 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a., hostname=jenkins-hbase4.apache.org,38905,1681314818739, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-12 15:55:27,870 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/A/bd87a964601b4b7caa7f139b5fb0af53 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/bd87a964601b4b7caa7f139b5fb0af53 2023-04-12 15:55:27,874 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in d7d8c2ad266b62b02ebd2f4b9a678b1a/A of d7d8c2ad266b62b02ebd2f4b9a678b1a into bd87a964601b4b7caa7f139b5fb0af53(size=25.7 K), total size for store is 25.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:55:27,874 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d7d8c2ad266b62b02ebd2f4b9a678b1a: 2023-04-12 15:55:27,874 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a., storeName=d7d8c2ad266b62b02ebd2f4b9a678b1a/A, priority=10, startTime=1681314927004; duration=0sec 2023-04-12 15:55:27,874 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:55:27,879 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/C/63dd40f4fd6b4049b74c5e05ff33a4d0 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/63dd40f4fd6b4049b74c5e05ff33a4d0 2023-04-12 15:55:27,883 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in d7d8c2ad266b62b02ebd2f4b9a678b1a/C of d7d8c2ad266b62b02ebd2f4b9a678b1a into 63dd40f4fd6b4049b74c5e05ff33a4d0(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:55:27,883 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d7d8c2ad266b62b02ebd2f4b9a678b1a: 2023-04-12 15:55:27,883 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a., storeName=d7d8c2ad266b62b02ebd2f4b9a678b1a/C, priority=10, startTime=1681314927005; duration=0sec 2023-04-12 15:55:27,883 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:55:27,884 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:27,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59804 deadline: 1681314987880, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:27,890 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:27,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59810 deadline: 1681314987884, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:27,927 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=198 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/B/439698319a144188a6561d466b958ff5 2023-04-12 15:55:28,038 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:28,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59902 deadline: 1681314988036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:28,054 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:28,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59882 deadline: 1681314988051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:28,352 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=198 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/C/aca9c9254bb84b1a84050a262795610a 2023-04-12 15:55:28,357 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/A/6a67a5c9dc714fb8bae990f7bc098db7 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/6a67a5c9dc714fb8bae990f7bc098db7 2023-04-12 15:55:28,363 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/6a67a5c9dc714fb8bae990f7bc098db7, entries=150, sequenceid=198, filesize=25.3 K 2023-04-12 15:55:28,364 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/B/439698319a144188a6561d466b958ff5 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/439698319a144188a6561d466b958ff5 2023-04-12 15:55:28,369 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/439698319a144188a6561d466b958ff5, entries=150, sequenceid=198, filesize=11.8 K 2023-04-12 15:55:28,370 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/C/aca9c9254bb84b1a84050a262795610a as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/aca9c9254bb84b1a84050a262795610a 2023-04-12 15:55:28,381 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/aca9c9254bb84b1a84050a262795610a, entries=150, sequenceid=198, filesize=11.8 K 2023-04-12 15:55:28,382 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for d7d8c2ad266b62b02ebd2f4b9a678b1a in 1294ms, sequenceid=198, compaction requested=false 2023-04-12 15:55:28,382 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d7d8c2ad266b62b02ebd2f4b9a678b1a: 2023-04-12 15:55:28,382 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:28,382 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:28,382 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:28,382 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:28,382 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:28,382 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:28,384 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:28,384 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:28,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:28,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:28,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:28,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:28,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:28,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:28,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:28,385 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:28,386 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:28,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:28,386 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:28,386 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48b85c95[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:28,386 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:28,386 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:28,391 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:28,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:28,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:28,391 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:28,391 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:28,391 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:28,392 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:28,392 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:28,393 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:28,393 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:28,393 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:28,393 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:28,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:28,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:28,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:28,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:28,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:28,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:28,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:28,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:28,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:28,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:28,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:28,395 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:28,395 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:28,395 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:28,395 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:28,395 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:28,395 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:28,399 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:28,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:28,399 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:28,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:28,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:28,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:28,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:28,399 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:28,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:28,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:28,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:28,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:28,400 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:28,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:28,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:28,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:28,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:28,403 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:28,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:28,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:28,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:28,405 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:28,405 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:28,405 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:28,405 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:28,405 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:28,405 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:28,406 DEBUG [Thread-2432] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:28,406 DEBUG [Thread-2432] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:28,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:28,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:28,405 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:28,406 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:28,406 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:28,406 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:28,406 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:28,406 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:28,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:28,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:28,506 DEBUG [Thread-2432] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:28,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:28,508 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:28,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:28,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:28,509 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:28,509 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:28,509 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:28,509 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:28,509 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:28,509 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:28,511 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:28,511 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:28,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:28,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:28,512 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:28,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:28,512 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:28,512 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:28,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:28,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:28,512 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:28,513 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:28,513 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:28,513 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:28,513 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:28,513 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:28,513 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:28,513 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:28,513 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:28,513 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. started... 2023-04-12 15:55:28,513 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d7d8c2ad266b62b02ebd2f4b9a678b1a 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-12 15:55:28,513 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7d8c2ad266b62b02ebd2f4b9a678b1a, store=A 2023-04-12 15:55:28,513 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:28,513 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7d8c2ad266b62b02ebd2f4b9a678b1a, store=B 2023-04-12 15:55:28,513 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:28,513 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7d8c2ad266b62b02ebd2f4b9a678b1a, store=C 2023-04-12 15:55:28,513 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:28,526 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304127fff971a256446758a82385ae5948185 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304127fff971a256446758a82385ae5948185 2023-04-12 15:55:28,531 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=219, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/A/2484a203716e4f4b96e33c5050ea7aa1 2023-04-12 15:55:28,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on d7d8c2ad266b62b02ebd2f4b9a678b1a 2023-04-12 15:55:28,547 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. as already flushing 2023-04-12 15:55:28,552 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=219 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/B/c402bb47e31643fbb80a0e8dd81c7a13 2023-04-12 15:55:28,562 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=219 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/C/189d5f660cd845c0a72abe3aa3ce7200 2023-04-12 15:55:28,566 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/A/2484a203716e4f4b96e33c5050ea7aa1 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/2484a203716e4f4b96e33c5050ea7aa1 2023-04-12 15:55:28,569 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/2484a203716e4f4b96e33c5050ea7aa1, entries=150, sequenceid=219, filesize=25.3 K 2023-04-12 15:55:28,570 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/B/c402bb47e31643fbb80a0e8dd81c7a13 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/c402bb47e31643fbb80a0e8dd81c7a13 2023-04-12 15:55:28,573 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/c402bb47e31643fbb80a0e8dd81c7a13, entries=150, sequenceid=219, filesize=11.8 K 2023-04-12 15:55:28,573 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/C/189d5f660cd845c0a72abe3aa3ce7200 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/189d5f660cd845c0a72abe3aa3ce7200 2023-04-12 15:55:28,576 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/189d5f660cd845c0a72abe3aa3ce7200, entries=150, sequenceid=219, filesize=11.8 K 2023-04-12 15:55:28,577 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=60.38 KB/61830 for d7d8c2ad266b62b02ebd2f4b9a678b1a in 64ms, sequenceid=219, compaction requested=true 2023-04-12 15:55:28,577 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d7d8c2ad266b62b02ebd2f4b9a678b1a: 2023-04-12 15:55:28,577 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:28,577 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:28,577 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:28,577 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:28,577 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:28,577 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:28,578 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:28,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on d7d8c2ad266b62b02ebd2f4b9a678b1a 2023-04-12 15:55:28,578 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d7d8c2ad266b62b02ebd2f4b9a678b1a 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-12 15:55:28,578 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:28,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:28,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:28,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:28,579 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7d8c2ad266b62b02ebd2f4b9a678b1a, store=A 2023-04-12 15:55:28,579 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:28,579 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7d8c2ad266b62b02ebd2f4b9a678b1a, store=B 2023-04-12 15:55:28,579 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:28,579 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:28,579 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:28,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:28,579 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7d8c2ad266b62b02ebd2f4b9a678b1a, store=C 2023-04-12 15:55:28,579 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:28,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:28,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:28,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:28,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:28,588 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:28,588 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5943e24d[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:28,588 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:28,588 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:28,590 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:28,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:28,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:28,590 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:28,590 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:28,590 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:28,590 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:28,590 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:28,591 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:28,591 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:28,591 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:28,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:28,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:28,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:28,591 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:28,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:28,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:28,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:28,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:28,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:28,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:28,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:28,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:28,594 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:28,594 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:28,594 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:28,594 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:28,594 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:28,594 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:28,597 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:28,597 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:28,597 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:28,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:28,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:28,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:28,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:28,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:28,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:28,597 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:28,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:28,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:28,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:28,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:28,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:28,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:28,598 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:28,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:28,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:28,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:28,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:28,601 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:28,601 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:28,601 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:28,601 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:28,601 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:28,601 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:28,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:28,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:28,601 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:28,601 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:28,601 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:28,601 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:28,601 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:28,601 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:28,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:28,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:28,601 DEBUG [Thread-2432] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:28,601 DEBUG [Thread-2432] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:28,614 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230412979dd48861c84db0b18208c62dc7b2cc to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412979dd48861c84db0b18208c62dc7b2cc 2023-04-12 15:55:28,643 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:28,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59882 deadline: 1681314988632, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:28,643 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:28,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59902 deadline: 1681314988633, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:28,702 DEBUG [Thread-2432] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:28,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:28,709 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:28,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:28,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:28,716 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:28,716 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:28,716 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:28,716 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:28,717 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:28,717 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:28,718 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:28,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:28,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:28,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:28,719 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:28,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:28,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:28,719 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:28,719 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:28,720 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:28,720 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:28,720 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:28,720 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:28,720 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:28,721 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:28,721 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:28,730 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:28,730 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:28,730 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:28,730 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. started... 2023-04-12 15:55:28,730 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. as already flushing 2023-04-12 15:55:28,730 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:28,730 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:28,730 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:28,730 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:28,730 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:28,730 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:28,732 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:28,732 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:28,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:28,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:28,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:28,732 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:28,732 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:28,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:28,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:28,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:28,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:28,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:28,734 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:28,734 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5712ed48[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:28,734 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:28,734 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:28,736 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:28,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:28,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:28,736 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:28,736 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:28,736 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:28,736 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:28,736 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:28,738 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:28,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:28,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:28,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:28,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:28,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:28,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:28,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:28,739 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:28,739 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:28,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:28,739 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:28,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:28,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:28,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:28,741 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:28,741 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:28,741 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:28,741 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:28,741 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:28,741 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:28,742 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:28,742 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:28,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:28,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:28,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:28,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:28,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:28,742 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:28,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:28,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:28,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:28,743 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:28,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:28,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:28,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:28,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:28,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:28,744 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:28,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:28,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:28,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:28,750 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:28,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59882 deadline: 1681314988745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:28,750 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:28,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59902 deadline: 1681314988746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:28,751 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:28,751 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:28,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:28,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:28,751 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:28,751 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:28,751 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:28,751 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:28,752 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:28,752 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:28,752 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:28,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:28,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:28,752 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:28,752 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:28,752 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:28,752 DEBUG [Thread-2432] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:28,752 DEBUG [Thread-2432] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:28,852 DEBUG [Thread-2432] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:28,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:28,855 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:28,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:28,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:28,858 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:28,858 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:28,858 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:28,858 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:28,859 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:28,859 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:28,860 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:28,860 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:28,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:28,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:28,860 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:28,860 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:28,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:28,861 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:28,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:28,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:28,861 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:28,862 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:28,862 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:28,862 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:28,862 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:28,862 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:28,862 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:28,862 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:28,862 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:28,862 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. started... 2023-04-12 15:55:28,862 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. as already flushing 2023-04-12 15:55:28,862 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:28,862 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:28,862 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:28,862 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:28,862 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:28,862 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:28,864 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:28,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:28,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:28,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:28,864 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:28,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:28,865 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:28,865 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:28,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:28,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:28,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:28,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:28,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:28,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d38eeda[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:28,866 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:28,866 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:28,867 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:28,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:28,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:28,867 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:28,868 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:28,868 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:28,868 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:28,868 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:28,870 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:28,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:28,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:28,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:28,871 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:28,871 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:28,871 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:28,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:28,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:28,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:28,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:28,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:28,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:28,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:28,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:28,873 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:28,873 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:28,874 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:28,874 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:28,874 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:28,874 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:28,875 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:28,875 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:28,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:28,875 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:28,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:28,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:28,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:28,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:28,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:28,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:28,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:28,879 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:28,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:28,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:28,882 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:28,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:28,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:28,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:28,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:28,884 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:28,884 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:28,884 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:28,884 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:28,884 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:28,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:28,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:28,884 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:28,884 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:28,885 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:28,885 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:28,884 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:28,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:28,885 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:28,884 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:28,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:28,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:28,885 DEBUG [Thread-2432] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:28,885 DEBUG [Thread-2432] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:28,958 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:28,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59882 deadline: 1681314988951, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:28,959 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:28,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59902 deadline: 1681314988952, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:28,985 DEBUG [Thread-2432] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:28,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:28,989 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:28,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:28,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:28,990 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:28,990 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:28,990 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:28,990 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:28,990 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:28,990 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:28,992 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:28,992 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:28,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:28,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:28,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:28,993 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:28,993 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:28,993 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:28,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:28,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:28,994 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:28,996 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:28,996 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:28,996 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:28,996 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:28,996 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:29,004 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:29,004 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:29,004 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:29,004 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. started... 2023-04-12 15:55:29,004 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. as already flushing 2023-04-12 15:55:29,004 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:29,004 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:29,004 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:29,004 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:29,004 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:29,004 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:29,007 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:29,007 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:29,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:29,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:29,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:29,008 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:29,008 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:29,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:29,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:29,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:29,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:29,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:29,013 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:29,013 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ee471c[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:29,013 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:29,013 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:29,015 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:29,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:29,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:29,015 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:29,015 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:29,015 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:29,015 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:29,015 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:29,017 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:29,017 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:29,017 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:29,019 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:29,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:29,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:29,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:29,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:29,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:29,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:29,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:29,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:29,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:29,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:29,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:29,023 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:29,023 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:29,023 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:29,023 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:29,023 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:29,023 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:29,025 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:29,025 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:29,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:29,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:29,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:29,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:29,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:29,025 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:29,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:29,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:29,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:29,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:29,027 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:29,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:29,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:29,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:29,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:29,029 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:29,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:29,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:29,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:29,030 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=232, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/A/bb1d6b402f6646aaaad1bde2a3eb29ea 2023-04-12 15:55:29,033 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:29,033 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:29,033 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:29,033 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:29,033 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:29,033 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:29,034 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:29,034 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:29,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:29,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:29,033 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:29,034 DEBUG [Thread-2432] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:29,034 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:29,034 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:29,034 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:29,034 DEBUG [Thread-2432] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:29,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:29,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:29,051 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=232 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/B/938ee0d200b34f27becf6607a485f098 2023-04-12 15:55:29,081 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=232 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/C/93ed31725fca4baab843b39315bc5927 2023-04-12 15:55:29,087 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/A/bb1d6b402f6646aaaad1bde2a3eb29ea as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/bb1d6b402f6646aaaad1bde2a3eb29ea 2023-04-12 15:55:29,092 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/bb1d6b402f6646aaaad1bde2a3eb29ea, entries=250, sequenceid=232, filesize=38.9 K 2023-04-12 15:55:29,094 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/B/938ee0d200b34f27becf6607a485f098 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/938ee0d200b34f27becf6607a485f098 2023-04-12 15:55:29,102 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/938ee0d200b34f27becf6607a485f098, entries=150, sequenceid=232, filesize=11.8 K 2023-04-12 15:55:29,104 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/C/93ed31725fca4baab843b39315bc5927 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/93ed31725fca4baab843b39315bc5927 2023-04-12 15:55:29,110 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/93ed31725fca4baab843b39315bc5927, entries=150, sequenceid=232, filesize=11.8 K 2023-04-12 15:55:29,111 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for d7d8c2ad266b62b02ebd2f4b9a678b1a in 533ms, sequenceid=232, compaction requested=true 2023-04-12 15:55:29,111 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d7d8c2ad266b62b02ebd2f4b9a678b1a: 2023-04-12 15:55:29,111 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:55:29,111 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-12 15:55:29,111 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-12 15:55:29,111 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:55:29,111 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:55:29,112 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48554 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-12 15:55:29,112 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1912): d7d8c2ad266b62b02ebd2f4b9a678b1a/B is initiating minor compaction (all files) 2023-04-12 15:55:29,112 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d7d8c2ad266b62b02ebd2f4b9a678b1a/B in TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:29,113 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/76b6b15d792c4cd9aca15b9dd77c1336, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/439698319a144188a6561d466b958ff5, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/c402bb47e31643fbb80a0e8dd81c7a13, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/938ee0d200b34f27becf6607a485f098] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp, totalSize=47.4 K 2023-04-12 15:55:29,113 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 117978 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-12 15:55:29,113 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1912): d7d8c2ad266b62b02ebd2f4b9a678b1a/A is initiating minor compaction (all files) 2023-04-12 15:55:29,113 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d7d8c2ad266b62b02ebd2f4b9a678b1a/A in TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:29,113 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/bd87a964601b4b7caa7f139b5fb0af53, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/6a67a5c9dc714fb8bae990f7bc098db7, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/2484a203716e4f4b96e33c5050ea7aa1, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/bb1d6b402f6646aaaad1bde2a3eb29ea] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp, totalSize=115.2 K 2023-04-12 15:55:29,113 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 76b6b15d792c4cd9aca15b9dd77c1336, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=184, earliestPutTs=1681314925176 2023-04-12 15:55:29,113 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting bd87a964601b4b7caa7f139b5fb0af53, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=184, earliestPutTs=1681314925176 2023-04-12 15:55:29,114 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 439698319a144188a6561d466b958ff5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=198, earliestPutTs=1681314925728 2023-04-12 15:55:29,115 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 6a67a5c9dc714fb8bae990f7bc098db7, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=198, earliestPutTs=1681314925728 2023-04-12 15:55:29,115 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting c402bb47e31643fbb80a0e8dd81c7a13, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=219, earliestPutTs=1681314927398 2023-04-12 15:55:29,115 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 2484a203716e4f4b96e33c5050ea7aa1, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=219, earliestPutTs=1681314927398 2023-04-12 15:55:29,115 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 938ee0d200b34f27becf6607a485f098, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=232, earliestPutTs=1681314928567 2023-04-12 15:55:29,115 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting bb1d6b402f6646aaaad1bde2a3eb29ea, keycount=250, bloomtype=ROW, size=38.9 K, encoding=NONE, compression=NONE, seqNum=232, earliestPutTs=1681314928561 2023-04-12 15:55:29,123 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] throttle.PressureAwareThroughputController(145): d7d8c2ad266b62b02ebd2f4b9a678b1a#B#compaction#365 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:55:29,134 DEBUG [Thread-2432] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:29,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:29,136 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:29,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:29,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:29,148 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:29,148 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:29,148 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:29,148 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:29,150 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:29,150 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:29,151 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:29,151 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:29,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:29,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:29,152 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:29,152 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:29,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:29,154 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:29,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:29,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:29,154 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:29,164 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:29,164 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:29,164 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:29,164 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:29,164 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:29,175 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] throttle.PressureAwareThroughputController(145): d7d8c2ad266b62b02ebd2f4b9a678b1a#A#compaction#366 average throughput is 0.38 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:55:29,180 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:29,180 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:29,180 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:29,180 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. started... 2023-04-12 15:55:29,180 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d7d8c2ad266b62b02ebd2f4b9a678b1a 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-12 15:55:29,181 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7d8c2ad266b62b02ebd2f4b9a678b1a, store=A 2023-04-12 15:55:29,181 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:29,181 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7d8c2ad266b62b02ebd2f4b9a678b1a, store=B 2023-04-12 15:55:29,181 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:29,181 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7d8c2ad266b62b02ebd2f4b9a678b1a, store=C 2023-04-12 15:55:29,181 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:29,232 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230412b2a105df94c240f4b1a88930d71df0d4 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412b2a105df94c240f4b1a88930d71df0d4 2023-04-12 15:55:29,255 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=251, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/A/fe3f3d0e2bc24fa283df4bf8230c13af 2023-04-12 15:55:29,266 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=251 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/B/3e19d4e27579461eada55849c5cb3a3b 2023-04-12 15:55:29,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on d7d8c2ad266b62b02ebd2f4b9a678b1a 2023-04-12 15:55:29,268 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. as already flushing 2023-04-12 15:55:29,316 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:29,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59882 deadline: 1681314989301, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:29,327 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:29,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59902 deadline: 1681314989316, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:29,425 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:29,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59882 deadline: 1681314989417, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:29,438 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:29,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59902 deadline: 1681314989429, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:29,570 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/B/89a61f9dcfec467a921eb25e7ee1f656 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/89a61f9dcfec467a921eb25e7ee1f656 2023-04-12 15:55:29,575 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d7d8c2ad266b62b02ebd2f4b9a678b1a/B of d7d8c2ad266b62b02ebd2f4b9a678b1a into 89a61f9dcfec467a921eb25e7ee1f656(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:55:29,575 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d7d8c2ad266b62b02ebd2f4b9a678b1a: 2023-04-12 15:55:29,575 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a., storeName=d7d8c2ad266b62b02ebd2f4b9a678b1a/B, priority=12, startTime=1681314929111; duration=0sec 2023-04-12 15:55:29,575 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:55:29,575 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-12 15:55:29,577 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48554 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-12 15:55:29,577 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1912): d7d8c2ad266b62b02ebd2f4b9a678b1a/C is initiating minor compaction (all files) 2023-04-12 15:55:29,577 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d7d8c2ad266b62b02ebd2f4b9a678b1a/C in TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:29,577 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/63dd40f4fd6b4049b74c5e05ff33a4d0, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/aca9c9254bb84b1a84050a262795610a, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/189d5f660cd845c0a72abe3aa3ce7200, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/93ed31725fca4baab843b39315bc5927] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp, totalSize=47.4 K 2023-04-12 15:55:29,577 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 63dd40f4fd6b4049b74c5e05ff33a4d0, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=184, earliestPutTs=1681314925176 2023-04-12 15:55:29,578 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting aca9c9254bb84b1a84050a262795610a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=198, earliestPutTs=1681314925728 2023-04-12 15:55:29,578 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 189d5f660cd845c0a72abe3aa3ce7200, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=219, earliestPutTs=1681314927398 2023-04-12 15:55:29,578 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 93ed31725fca4baab843b39315bc5927, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=232, earliestPutTs=1681314928567 2023-04-12 15:55:29,586 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] throttle.PressureAwareThroughputController(145): d7d8c2ad266b62b02ebd2f4b9a678b1a#C#compaction#370 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:55:29,628 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:29,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59882 deadline: 1681314989626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:29,645 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:29,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59902 deadline: 1681314989641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:29,703 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=251 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/C/b568c95af0734d8cb036046fc3346665 2023-04-12 15:55:29,712 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/A/fe3f3d0e2bc24fa283df4bf8230c13af as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/fe3f3d0e2bc24fa283df4bf8230c13af 2023-04-12 15:55:29,716 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/fe3f3d0e2bc24fa283df4bf8230c13af, entries=150, sequenceid=251, filesize=25.3 K 2023-04-12 15:55:29,716 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/B/3e19d4e27579461eada55849c5cb3a3b as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/3e19d4e27579461eada55849c5cb3a3b 2023-04-12 15:55:29,720 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/3e19d4e27579461eada55849c5cb3a3b, entries=150, sequenceid=251, filesize=11.8 K 2023-04-12 15:55:29,721 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/C/b568c95af0734d8cb036046fc3346665 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/b568c95af0734d8cb036046fc3346665 2023-04-12 15:55:29,724 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/b568c95af0734d8cb036046fc3346665, entries=150, sequenceid=251, filesize=11.8 K 2023-04-12 15:55:29,725 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for d7d8c2ad266b62b02ebd2f4b9a678b1a in 545ms, sequenceid=251, compaction requested=false 2023-04-12 15:55:29,725 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d7d8c2ad266b62b02ebd2f4b9a678b1a: 2023-04-12 15:55:29,725 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:29,725 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:29,725 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:29,725 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:29,725 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:29,725 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:29,727 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:29,727 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:29,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:29,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:29,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:29,727 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:29,727 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:29,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:29,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:29,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:29,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:29,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:29,729 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:29,729 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41f98c26[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:29,729 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:29,729 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:29,730 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:29,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:29,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:29,731 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:29,731 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:29,731 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:29,731 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:29,731 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:29,733 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:29,734 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:29,734 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:29,734 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:29,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:29,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:29,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:29,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:29,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:29,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:29,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:29,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:29,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:29,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:29,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:29,738 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:29,738 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:29,738 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:29,738 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:29,738 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:29,738 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:29,740 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:29,740 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:29,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:29,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:29,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:29,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:29,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:29,740 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:29,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:29,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:29,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:29,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:29,742 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:29,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:29,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:29,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:29,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:29,744 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:29,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:29,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:29,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:29,752 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:29,752 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:29,752 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:29,752 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:29,752 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:29,752 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:29,752 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:29,752 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:29,753 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:29,752 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:29,753 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:29,752 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:29,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:29,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:29,753 DEBUG [Thread-2432] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:29,753 DEBUG [Thread-2432] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:29,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:29,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:29,853 DEBUG [Thread-2432] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:29,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:29,855 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:29,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:29,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:29,856 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:29,856 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:29,856 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:29,856 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:29,857 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:29,857 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:29,859 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:29,859 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:29,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:29,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:29,860 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:29,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:29,860 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:29,860 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:29,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:29,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:29,861 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:29,861 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:29,861 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:29,861 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:29,861 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:29,861 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:29,861 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:29,861 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:29,861 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:29,861 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. started... 2023-04-12 15:55:29,862 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d7d8c2ad266b62b02ebd2f4b9a678b1a 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-12 15:55:29,862 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7d8c2ad266b62b02ebd2f4b9a678b1a, store=A 2023-04-12 15:55:29,862 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:29,862 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7d8c2ad266b62b02ebd2f4b9a678b1a, store=B 2023-04-12 15:55:29,862 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:29,862 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7d8c2ad266b62b02ebd2f4b9a678b1a, store=C 2023-04-12 15:55:29,862 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:29,872 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041208d61b523f994a719ecc5df20453abdb to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041208d61b523f994a719ecc5df20453abdb 2023-04-12 15:55:29,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on d7d8c2ad266b62b02ebd2f4b9a678b1a 2023-04-12 15:55:29,900 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. as already flushing 2023-04-12 15:55:29,955 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:29,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59902 deadline: 1681314989948, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:29,957 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:29,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59810 deadline: 1681314989948, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:29,957 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:29,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59804 deadline: 1681314989949, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:29,961 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:29,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59882 deadline: 1681314989955, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:30,002 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/C/5decac26913342389fd06c008c80097a as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/5decac26913342389fd06c008c80097a 2023-04-12 15:55:30,007 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d7d8c2ad266b62b02ebd2f4b9a678b1a/C of d7d8c2ad266b62b02ebd2f4b9a678b1a into 5decac26913342389fd06c008c80097a(size=12.3 K), total size for store is 24.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:55:30,007 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d7d8c2ad266b62b02ebd2f4b9a678b1a: 2023-04-12 15:55:30,007 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a., storeName=d7d8c2ad266b62b02ebd2f4b9a678b1a/C, priority=12, startTime=1681314929111; duration=0sec 2023-04-12 15:55:30,008 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:55:30,064 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:30,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59810 deadline: 1681314990058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:30,064 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:30,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59804 deadline: 1681314990058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:30,064 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:30,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59882 deadline: 1681314990062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:30,268 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:30,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59810 deadline: 1681314990265, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:30,269 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:30,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59882 deadline: 1681314990265, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:30,271 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:30,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59804 deadline: 1681314990266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:30,276 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=265, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/A/50714ded97514e99be8da3c958b8a6fc 2023-04-12 15:55:30,418 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/A/da4fce448dce450987a91a7ed8dd7054 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/da4fce448dce450987a91a7ed8dd7054 2023-04-12 15:55:30,422 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d7d8c2ad266b62b02ebd2f4b9a678b1a/A of d7d8c2ad266b62b02ebd2f4b9a678b1a into da4fce448dce450987a91a7ed8dd7054(size=25.8 K), total size for store is 51.1 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-12 15:55:30,422 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d7d8c2ad266b62b02ebd2f4b9a678b1a: 2023-04-12 15:55:30,422 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a., storeName=d7d8c2ad266b62b02ebd2f4b9a678b1a/A, priority=12, startTime=1681314929111; duration=1sec 2023-04-12 15:55:30,422 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:55:30,460 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:30,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59902 deadline: 1681314990456, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:30,576 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:30,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59810 deadline: 1681314990569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:30,577 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:30,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59882 deadline: 1681314990571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:30,577 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:30,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59804 deadline: 1681314990572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:30,688 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=265 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/B/116bd8356ff34924999e86daf7547bbe 2023-04-12 15:55:31,084 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:31,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59810 deadline: 1681314991081, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:31,084 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:31,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59882 deadline: 1681314991081, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:31,084 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:31,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59804 deadline: 1681314991082, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:31,123 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=265 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/C/bf8e37a5850d4275a01ca80f8b02939a 2023-04-12 15:55:31,127 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/A/50714ded97514e99be8da3c958b8a6fc as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/50714ded97514e99be8da3c958b8a6fc 2023-04-12 15:55:31,132 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/50714ded97514e99be8da3c958b8a6fc, entries=150, sequenceid=265, filesize=25.4 K 2023-04-12 15:55:31,133 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/B/116bd8356ff34924999e86daf7547bbe as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/116bd8356ff34924999e86daf7547bbe 2023-04-12 15:55:31,138 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/116bd8356ff34924999e86daf7547bbe, entries=150, sequenceid=265, filesize=11.9 K 2023-04-12 15:55:31,139 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/C/bf8e37a5850d4275a01ca80f8b02939a as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/bf8e37a5850d4275a01ca80f8b02939a 2023-04-12 15:55:31,146 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/bf8e37a5850d4275a01ca80f8b02939a, entries=150, sequenceid=265, filesize=11.9 K 2023-04-12 15:55:31,147 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for d7d8c2ad266b62b02ebd2f4b9a678b1a in 1286ms, sequenceid=265, compaction requested=true 2023-04-12 15:55:31,147 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d7d8c2ad266b62b02ebd2f4b9a678b1a: 2023-04-12 15:55:31,147 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:31,147 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:31,147 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:31,147 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:31,147 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:31,147 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:31,148 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:31,149 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:31,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:31,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:31,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:31,149 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:31,149 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:31,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:31,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:31,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:31,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:31,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:31,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:31,153 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c141fc[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:31,153 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:31,153 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:31,155 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:31,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:31,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:31,155 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:31,155 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:31,155 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:31,155 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:31,156 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:31,158 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:31,158 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:31,158 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:31,158 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:31,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:31,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:31,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:31,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:31,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:31,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:31,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:31,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:31,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:31,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:31,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:31,172 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:31,172 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:31,172 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:31,172 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:31,172 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:31,172 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:31,174 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:31,174 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:31,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:31,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:31,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:31,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:31,174 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:31,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:31,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:31,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:31,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:31,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:31,175 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:31,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:31,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:31,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:31,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:31,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:31,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:31,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:31,180 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:31,183 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:31,183 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:31,183 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:31,183 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:31,183 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:31,184 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:31,184 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:31,184 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:31,184 DEBUG [Thread-2432] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:31,184 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:31,184 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:31,184 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:31,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:31,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:31,184 DEBUG [Thread-2432] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:31,184 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:31,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:31,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:31,284 DEBUG [Thread-2432] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:31,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:31,286 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:31,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:31,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:31,288 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:31,288 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:31,288 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:31,288 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:31,288 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:31,288 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:31,289 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:31,289 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:31,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:31,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:31,290 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:31,290 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:31,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:31,290 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:31,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:31,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:31,290 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:31,291 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:31,291 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:31,291 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:31,291 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:31,291 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:31,291 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:31,291 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:31,291 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:31,291 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. started... 2023-04-12 15:55:31,291 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d7d8c2ad266b62b02ebd2f4b9a678b1a 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-12 15:55:31,291 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7d8c2ad266b62b02ebd2f4b9a678b1a, store=A 2023-04-12 15:55:31,292 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:31,292 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7d8c2ad266b62b02ebd2f4b9a678b1a, store=B 2023-04-12 15:55:31,292 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:31,292 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7d8c2ad266b62b02ebd2f4b9a678b1a, store=C 2023-04-12 15:55:31,292 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:31,307 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304129a9e3e05f48f4d838f48c47e1c6ccb45 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304129a9e3e05f48f4d838f48c47e1c6ccb45 2023-04-12 15:55:31,311 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=286, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/A/c9146fc2043c482c9522d42863f16176 2023-04-12 15:55:31,322 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=286 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/B/7a138e876f0247b4a2623c6fd91c8cd8 2023-04-12 15:55:31,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on d7d8c2ad266b62b02ebd2f4b9a678b1a 2023-04-12 15:55:31,467 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. as already flushing 2023-04-12 15:55:31,529 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:31,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59902 deadline: 1681314991521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:31,635 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:31,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59902 deadline: 1681314991631, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:31,746 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=286 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/C/a0ee708199a44bfc9e33c2bab40ef921 2023-04-12 15:55:31,751 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/A/c9146fc2043c482c9522d42863f16176 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/c9146fc2043c482c9522d42863f16176 2023-04-12 15:55:31,755 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/c9146fc2043c482c9522d42863f16176, entries=150, sequenceid=286, filesize=25.4 K 2023-04-12 15:55:31,756 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/B/7a138e876f0247b4a2623c6fd91c8cd8 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/7a138e876f0247b4a2623c6fd91c8cd8 2023-04-12 15:55:31,760 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/7a138e876f0247b4a2623c6fd91c8cd8, entries=150, sequenceid=286, filesize=11.9 K 2023-04-12 15:55:31,761 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/C/a0ee708199a44bfc9e33c2bab40ef921 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/a0ee708199a44bfc9e33c2bab40ef921 2023-04-12 15:55:31,765 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/a0ee708199a44bfc9e33c2bab40ef921, entries=150, sequenceid=286, filesize=11.9 K 2023-04-12 15:55:31,766 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for d7d8c2ad266b62b02ebd2f4b9a678b1a in 475ms, sequenceid=286, compaction requested=true 2023-04-12 15:55:31,766 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d7d8c2ad266b62b02ebd2f4b9a678b1a: 2023-04-12 15:55:31,766 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:31,766 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:31,766 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:31,767 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:31,767 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:31,767 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:31,768 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:31,768 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:31,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:31,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:31,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:31,769 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:31,769 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:31,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:31,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:31,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:31,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:31,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:31,770 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:31,770 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b452120[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:31,770 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:31,770 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:31,772 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:31,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:31,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:31,772 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:31,772 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:31,772 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:31,772 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:31,772 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:31,774 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:31,774 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:31,774 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:31,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:31,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:31,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:31,774 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:31,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:31,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:31,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:31,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:31,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:31,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:31,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:31,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:31,777 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:31,777 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:31,777 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:31,777 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:31,777 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:31,777 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:31,779 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:31,779 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:31,779 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:31,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:31,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:31,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:31,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:31,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:31,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:31,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:31,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:31,780 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:31,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:31,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:31,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:31,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:31,781 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:31,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:31,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:31,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:31,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:31,784 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:31,784 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:31,784 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:31,784 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:31,784 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:31,784 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:31,784 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:31,784 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:31,784 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:31,784 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:31,784 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:31,785 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:31,785 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:31,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:31,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:31,785 DEBUG [Thread-2432] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:31,785 DEBUG [Thread-2432] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:31,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:31,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:31,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on d7d8c2ad266b62b02ebd2f4b9a678b1a 2023-04-12 15:55:31,844 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d7d8c2ad266b62b02ebd2f4b9a678b1a 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-12 15:55:31,844 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7d8c2ad266b62b02ebd2f4b9a678b1a, store=A 2023-04-12 15:55:31,844 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:31,844 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7d8c2ad266b62b02ebd2f4b9a678b1a, store=B 2023-04-12 15:55:31,844 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:31,845 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7d8c2ad266b62b02ebd2f4b9a678b1a, store=C 2023-04-12 15:55:31,845 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:31,861 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304124a686e237604451c812fe99aeb0e11d1 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304124a686e237604451c812fe99aeb0e11d1 2023-04-12 15:55:31,885 DEBUG [Thread-2432] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:31,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:31,887 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:31,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:31,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:31,891 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:31,891 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:31,891 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:31,891 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:31,891 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:31,892 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:31,893 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:31,893 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:31,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:31,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:31,893 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:31,893 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:31,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:31,893 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:31,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:31,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:31,894 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:31,894 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:31,894 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:31,894 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:31,894 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:31,894 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:31,895 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:31,895 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:31,895 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:31,895 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. started... 2023-04-12 15:55:31,895 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. as already flushing 2023-04-12 15:55:31,895 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:31,895 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:31,895 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:31,895 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:31,895 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:31,895 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:31,897 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:31,897 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:31,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:31,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:31,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:31,897 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:31,897 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:31,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:31,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:31,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:31,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:31,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:31,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:31,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e6659f3[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:31,899 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:31,899 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:31,900 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:31,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:31,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:31,900 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:31,900 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:31,900 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:31,900 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:31,900 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:31,902 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:31,902 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:31,902 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:31,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:31,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:31,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:31,902 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:31,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:31,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:31,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:31,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:31,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:31,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:31,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:31,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:31,905 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:31,905 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:31,906 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:31,906 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:31,906 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:31,906 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:31,908 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:31,908 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:31,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:31,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:31,908 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:31,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:31,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:31,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:31,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:31,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:31,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:31,908 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:31,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:31,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:31,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:31,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:31,909 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:31,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:31,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:31,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:31,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:31,912 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:31,912 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:31,912 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:31,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:31,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:31,912 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:31,912 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:31,912 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:31,912 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:31,912 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:31,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:31,912 DEBUG [Thread-2432] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:31,912 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:31,912 DEBUG [Thread-2432] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:31,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:31,913 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:31,913 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:31,913 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:31,942 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:31,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59902 deadline: 1681314991940, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,013 DEBUG [Thread-2432] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:32,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:32,014 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:32,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:32,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:32,016 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:32,016 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:32,016 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:32,016 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:32,016 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:32,016 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:32,018 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,018 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:32,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:32,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:32,019 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,019 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:32,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:32,019 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:32,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:32,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:32,019 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:32,020 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:32,020 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:32,020 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:32,020 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:32,020 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:32,020 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:32,020 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:32,020 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:32,020 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. started... 2023-04-12 15:55:32,020 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. as already flushing 2023-04-12 15:55:32,020 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:32,020 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:32,020 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:32,020 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,020 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:32,020 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:32,031 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,031 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:32,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:32,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:32,032 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:32,032 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:32,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:32,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:32,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:32,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:32,033 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:32,033 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33f36aa5[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:32,033 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:32,033 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:32,035 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:32,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:32,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:32,035 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,035 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:32,035 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:32,035 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:32,035 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:32,036 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:32,036 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:32,036 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,036 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:32,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:32,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:32,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:32,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:32,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:32,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:32,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:32,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:32,039 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:32,039 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:32,039 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:32,039 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:32,039 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:32,039 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:32,040 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:32,040 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:32,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:32,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:32,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:32,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:32,040 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:32,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:32,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:32,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:32,041 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:32,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:32,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:32,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:32,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:32,042 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:32,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:32,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,045 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:32,045 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:32,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:32,045 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:32,045 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:32,045 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:32,045 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:32,045 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:32,046 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:32,046 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,046 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:32,046 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:32,046 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:32,046 DEBUG [Thread-2432] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:32,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:32,046 DEBUG [Thread-2432] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:32,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:32,046 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:32,047 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:32,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59902 deadline: 1681314992043, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,088 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:32,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59804 deadline: 1681314992085, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,100 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:32,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59810 deadline: 1681314992092, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,100 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:32,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59882 deadline: 1681314992092, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,146 DEBUG [Thread-2432] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:32,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:32,148 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:32,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:32,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:32,154 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:32,154 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:32,154 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:32,154 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:32,154 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:32,154 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:32,156 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,156 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:32,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:32,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:32,156 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,156 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:32,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:32,157 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:32,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:32,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:32,158 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:32,158 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:32,158 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:32,158 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:32,158 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:32,158 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:32,160 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:32,160 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:32,160 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:32,160 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. started... 2023-04-12 15:55:32,160 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. as already flushing 2023-04-12 15:55:32,160 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:32,160 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:32,160 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:32,160 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,160 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:32,160 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:32,162 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:32,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:32,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:32,162 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:32,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:32,163 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:32,163 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:32,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:32,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:32,165 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:32,165 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38b61386[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:32,165 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:32,165 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:32,166 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:32,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:32,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:32,167 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:32,167 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:32,167 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:32,167 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,167 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:32,169 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:32,169 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:32,169 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:32,172 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:32,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:32,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:32,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:32,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:32,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:32,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:32,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:32,180 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:32,180 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:32,180 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:32,180 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:32,180 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:32,180 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:32,184 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:32,184 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:32,184 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:32,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:32,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:32,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:32,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:32,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:32,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:32,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:32,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:32,185 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:32,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:32,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:32,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:32,186 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:32,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:32,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,189 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,189 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:32,189 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:32,189 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:32,189 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:32,189 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:32,189 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:32,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:32,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:32,189 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:32,189 DEBUG [Thread-2432] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:32,189 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:32,189 DEBUG [Thread-2432] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:32,190 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,190 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:32,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:32,190 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:32,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:32,190 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:32,255 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:32,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59902 deadline: 1681314992248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,266 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=300, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/A/769bc65e5d2e4daa9d03ea643e36dc6e 2023-04-12 15:55:32,290 DEBUG [Thread-2432] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:32,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:32,293 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:32,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:32,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:32,303 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:32,303 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:32,303 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:32,303 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:32,303 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:32,304 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:32,305 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,305 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:32,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:32,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:32,305 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,305 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:32,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:32,306 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:32,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:32,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:32,306 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:32,308 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:32,308 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:32,308 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:32,308 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:32,308 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:32,312 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:32,312 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:32,312 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:32,312 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. started... 2023-04-12 15:55:32,312 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. as already flushing 2023-04-12 15:55:32,312 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:32,312 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:32,312 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:32,312 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,312 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:32,313 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:32,315 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:32,315 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:32,315 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:32,316 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:32,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:32,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:32,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:32,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:32,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:32,317 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:32,317 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a987991[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:32,317 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:32,317 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:32,318 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:32,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:32,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:32,319 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:32,319 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:32,319 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:32,319 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,319 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:32,320 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:32,320 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,320 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:32,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:32,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:32,320 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:32,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:32,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:32,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:32,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:32,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:32,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:32,323 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:32,323 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:32,323 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:32,323 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:32,323 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:32,323 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:32,324 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:32,324 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:32,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:32,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:32,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:32,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:32,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:32,324 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:32,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:32,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:32,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:32,325 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:32,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:32,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:32,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:32,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,326 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:32,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:32,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,331 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,331 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:32,331 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:32,331 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,331 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:32,331 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:32,331 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:32,331 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:32,332 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:32,332 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:32,332 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:32,332 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:32,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:32,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:32,333 DEBUG [Thread-2432] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:32,333 DEBUG [Thread-2432] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:32,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:32,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:32,433 DEBUG [Thread-2432] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:32,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:32,435 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:32,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:32,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:32,437 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:32,437 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:32,437 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:32,437 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:32,437 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:32,437 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:32,439 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:32,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:32,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:32,439 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,439 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,439 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:32,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:32,440 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:32,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:32,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:32,440 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:32,441 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:32,441 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:32,442 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:32,442 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:32,442 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:32,449 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:32,449 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:32,449 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:32,449 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. started... 2023-04-12 15:55:32,449 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. as already flushing 2023-04-12 15:55:32,450 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:32,450 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:32,450 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:32,450 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,450 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:32,450 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:32,452 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,452 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:32,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:32,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:32,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:32,452 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:32,452 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:32,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:32,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:32,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:32,455 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:32,455 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:32,455 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56bd1af0[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:32,456 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:32,457 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,457 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:32,458 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:32,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:32,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:32,458 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:32,459 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:32,459 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:32,464 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,464 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:32,464 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:32,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:32,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:32,464 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:32,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:32,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:32,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:32,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:32,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:32,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:32,467 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:32,467 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:32,467 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:32,467 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:32,468 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:32,468 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:32,469 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:32,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:32,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:32,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:32,469 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:32,469 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:32,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:32,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:32,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:32,470 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:32,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:32,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:32,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:32,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:32,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:32,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,471 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:32,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:32,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,474 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:32,474 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:32,474 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:32,474 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:32,474 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:32,474 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:32,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:32,474 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:32,474 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:32,474 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:32,474 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,474 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:32,474 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:32,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:32,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:32,474 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:32,474 DEBUG [Thread-2432] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:32,475 DEBUG [Thread-2432] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:32,565 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:32,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59902 deadline: 1681314992558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,575 DEBUG [Thread-2432] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:32,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:32,577 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:32,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:32,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:32,578 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:32,578 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:32,578 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:32,578 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:32,578 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:32,578 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:32,580 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:32,580 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:32,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:32,580 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,580 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:32,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:32,581 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:32,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:32,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:32,582 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:32,582 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:32,582 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:32,582 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:32,582 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:32,582 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:32,582 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:32,582 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:32,582 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:32,582 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. started... 2023-04-12 15:55:32,582 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. as already flushing 2023-04-12 15:55:32,582 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:32,582 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:32,582 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:32,582 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,583 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:32,583 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:32,585 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:32,585 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:32,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:32,586 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:32,586 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:32,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:32,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:32,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:32,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:32,587 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:32,587 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e33e4c0[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:32,587 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:32,587 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:32,589 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:32,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:32,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:32,589 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:32,589 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:32,589 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:32,590 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,590 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:32,591 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,591 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:32,591 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:32,591 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:32,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:32,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:32,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:32,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:32,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:32,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:32,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:32,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:32,595 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:32,595 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:32,595 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:32,595 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:32,595 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:32,595 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:32,597 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:32,597 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:32,597 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:32,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:32,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:32,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:32,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:32,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:32,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:32,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:32,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:32,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:32,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:32,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:32,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:32,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,599 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:32,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:32,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,601 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,603 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,604 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:32,604 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:32,604 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,604 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:32,604 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:32,604 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:32,604 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:32,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:32,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:32,605 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:32,605 DEBUG [Thread-2432] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:32,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:32,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:32,605 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:32,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:32,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:32,605 DEBUG [Thread-2432] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:32,683 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=300 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/B/3d00aa0b1734468f87b052ec78b67077 2023-04-12 15:55:32,706 DEBUG [Thread-2432] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:32,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:32,708 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:32,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:32,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:32,709 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:32,709 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:32,709 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:32,709 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:32,709 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:32,709 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:32,710 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:32,710 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:32,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:32,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:32,711 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,711 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:32,711 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:32,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:32,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:32,712 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:32,712 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:32,712 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:32,712 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:32,712 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:32,712 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:32,712 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:32,712 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:32,712 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:32,712 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. started... 2023-04-12 15:55:32,712 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. as already flushing 2023-04-12 15:55:32,712 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:32,712 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:32,712 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:32,713 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,713 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:32,713 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:32,714 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,714 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:32,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:32,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:32,714 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:32,714 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:32,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:32,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:32,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:32,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:32,715 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:32,715 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7691f2a3[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:32,715 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:32,716 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:32,717 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:32,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:32,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:32,717 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:32,717 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:32,717 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,717 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:32,717 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:32,719 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,719 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:32,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:32,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:32,719 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:32,720 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:32,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:32,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:32,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:32,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:32,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:32,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:32,722 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:32,722 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:32,722 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:32,722 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:32,722 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:32,722 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:32,723 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:32,723 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:32,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:32,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:32,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:32,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:32,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:32,723 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:32,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:32,724 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:32,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:32,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:32,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:32,725 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:32,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:32,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:32,727 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,727 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:32,727 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:32,727 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:32,727 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:32,727 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:32,727 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,727 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:32,727 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:32,727 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:32,727 DEBUG [Thread-2432] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:32,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:32,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:32,727 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:32,727 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:32,727 DEBUG [Thread-2432] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:32,728 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:32,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:32,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:32,828 DEBUG [Thread-2432] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:32,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:32,831 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:32,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:32,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:32,833 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:32,833 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:32,833 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:32,833 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:32,834 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:32,834 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:32,835 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:32,835 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:32,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:32,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:32,836 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,836 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:32,836 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:32,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:32,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:32,837 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:32,840 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:32,840 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:32,840 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:32,840 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:32,840 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:32,840 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:32,840 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:32,840 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:32,840 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. started... 2023-04-12 15:55:32,841 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. as already flushing 2023-04-12 15:55:32,841 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:32,841 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:32,841 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:32,841 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,841 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:32,841 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:32,844 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,844 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:32,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:32,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:32,844 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:32,844 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:32,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:32,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:32,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:32,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:32,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:32,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f5c5911[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:32,845 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:32,845 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:32,846 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:32,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:32,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:32,846 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:32,847 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:32,847 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:32,847 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,847 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:32,848 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:32,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:32,849 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:32,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:32,849 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:32,849 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:32,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:32,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:32,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:32,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:32,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:32,851 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:32,851 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:32,851 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:32,851 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:32,851 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:32,851 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:32,856 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:32,856 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:32,856 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:32,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:32,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:32,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:32,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:32,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:32,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:32,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:32,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:32,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:32,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:32,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:32,861 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:32,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:32,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:32,862 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,865 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,865 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:32,865 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:32,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:32,865 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:32,865 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:32,865 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:32,865 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:32,865 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:32,865 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:32,865 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:32,865 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:32,865 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:32,865 DEBUG [Thread-2432] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:32,865 DEBUG [Thread-2432] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:32,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:32,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:32,965 DEBUG [Thread-2432] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:32,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:32,967 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:32,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:32,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:32,968 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:32,968 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:32,968 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:32,968 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:32,969 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:32,969 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:32,970 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:32,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:32,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:32,970 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:32,971 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,971 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:32,971 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:32,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:32,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:32,972 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:32,972 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:32,972 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:32,972 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:32,972 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:32,972 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:32,972 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:32,972 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:32,972 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:32,972 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. started... 2023-04-12 15:55:32,972 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. as already flushing 2023-04-12 15:55:32,972 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:32,972 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:32,972 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:32,973 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,973 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:32,973 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:32,974 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:32,974 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:32,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:32,974 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:32,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:32,975 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:32,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:32,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:32,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:32,976 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:32,976 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5108b490[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:32,976 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:32,976 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:32,977 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:32,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:32,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:32,978 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:32,978 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:32,978 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:32,978 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,979 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:32,979 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,979 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:32,980 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:32,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:32,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:32,980 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:32,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:32,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:32,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:32,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:32,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:32,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:32,982 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:32,982 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:32,982 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:32,982 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:32,982 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:32,982 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:32,983 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:32,983 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:32,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:32,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:32,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:32,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:32,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:32,983 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:32,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:32,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:32,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:32,984 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:32,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:32,985 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:32,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:32,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:32,987 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,987 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:32,987 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:32,987 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:32,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:32,987 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:32,987 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:32,987 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:32,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:32,987 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:32,987 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:32,987 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:32,987 DEBUG [Thread-2432] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:32,987 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:32,988 DEBUG [Thread-2432] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:32,988 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:32,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:32,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:32,988 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:33,073 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:33,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59902 deadline: 1681314993070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:33,088 DEBUG [Thread-2432] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:33,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:33,091 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:33,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:33,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:33,104 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:33,104 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:33,104 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:33,104 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:33,104 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:33,105 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:33,105 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=300 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/C/ac4fd241b7e24b2c8edf95cc91bd46cd 2023-04-12 15:55:33,107 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:33,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:33,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:33,107 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:33,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:33,107 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:33,107 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:33,108 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:33,109 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/A/769bc65e5d2e4daa9d03ea643e36dc6e as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/769bc65e5d2e4daa9d03ea643e36dc6e 2023-04-12 15:55:33,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:33,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:33,109 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:33,110 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:33,110 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:33,110 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:33,110 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:33,110 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:33,110 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:33,110 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:33,110 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:33,110 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. started... 2023-04-12 15:55:33,110 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. as already flushing 2023-04-12 15:55:33,110 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:33,110 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:33,110 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:33,111 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:33,111 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:33,111 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:33,112 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:33,112 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:33,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:33,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:33,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:33,113 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:33,113 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:33,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:33,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:33,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:33,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:33,115 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/769bc65e5d2e4daa9d03ea643e36dc6e, entries=200, sequenceid=300, filesize=32.3 K 2023-04-12 15:55:33,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:33,116 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:33,116 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/B/3d00aa0b1734468f87b052ec78b67077 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/3d00aa0b1734468f87b052ec78b67077 2023-04-12 15:55:33,116 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@307c5f95[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:33,116 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:33,116 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:33,118 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:33,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:33,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:33,118 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:33,118 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:33,118 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:33,118 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:33,118 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:33,120 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:33,120 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:33,120 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:33,120 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:33,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:33,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:33,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:33,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:33,121 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/3d00aa0b1734468f87b052ec78b67077, entries=150, sequenceid=300, filesize=11.9 K 2023-04-12 15:55:33,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:33,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:33,121 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/C/ac4fd241b7e24b2c8edf95cc91bd46cd as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/ac4fd241b7e24b2c8edf95cc91bd46cd 2023-04-12 15:55:33,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:33,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:33,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:33,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:33,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:33,123 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:33,123 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:33,123 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:33,123 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:33,123 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:33,124 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:33,126 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/ac4fd241b7e24b2c8edf95cc91bd46cd, entries=150, sequenceid=300, filesize=11.9 K 2023-04-12 15:55:33,127 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:33,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:33,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:33,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:33,127 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:33,127 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:33,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:33,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:33,127 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for d7d8c2ad266b62b02ebd2f4b9a678b1a in 1283ms, sequenceid=300, compaction requested=true 2023-04-12 15:55:33,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:33,127 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d7d8c2ad266b62b02ebd2f4b9a678b1a: 2023-04-12 15:55:33,127 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:55:33,127 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-12 15:55:33,127 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:33,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:33,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:33,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:33,127 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-12 15:55:33,127 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:55:33,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:33,128 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:55:33,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:33,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:33,129 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:33,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:33,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:33,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:33,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:33,131 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 137516 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-12 15:55:33,131 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1912): d7d8c2ad266b62b02ebd2f4b9a678b1a/A is initiating minor compaction (all files) 2023-04-12 15:55:33,132 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d7d8c2ad266b62b02ebd2f4b9a678b1a/A in TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:33,132 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/da4fce448dce450987a91a7ed8dd7054, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/fe3f3d0e2bc24fa283df4bf8230c13af, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/50714ded97514e99be8da3c958b8a6fc, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/c9146fc2043c482c9522d42863f16176, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/769bc65e5d2e4daa9d03ea643e36dc6e] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp, totalSize=134.3 K 2023-04-12 15:55:33,132 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61176 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-12 15:55:33,132 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:33,132 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1912): d7d8c2ad266b62b02ebd2f4b9a678b1a/B is initiating minor compaction (all files) 2023-04-12 15:55:33,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:33,132 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d7d8c2ad266b62b02ebd2f4b9a678b1a/B in TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:33,132 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/89a61f9dcfec467a921eb25e7ee1f656, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/3e19d4e27579461eada55849c5cb3a3b, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/116bd8356ff34924999e86daf7547bbe, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/7a138e876f0247b4a2623c6fd91c8cd8, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/3d00aa0b1734468f87b052ec78b67077] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp, totalSize=59.7 K 2023-04-12 15:55:33,132 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:33,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:33,132 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting da4fce448dce450987a91a7ed8dd7054, keycount=150, bloomtype=ROW, size=25.8 K, encoding=NONE, compression=NONE, seqNum=232, earliestPutTs=1681314928567 2023-04-12 15:55:33,133 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:33,133 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:33,133 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:33,133 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:33,133 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:33,133 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:33,133 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting fe3f3d0e2bc24fa283df4bf8230c13af, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=251, earliestPutTs=1681314928617 2023-04-12 15:55:33,133 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:33,133 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:33,133 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:33,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:33,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:33,133 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:33,133 DEBUG [Thread-2432] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:33,133 DEBUG [Thread-2432] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:33,133 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 50714ded97514e99be8da3c958b8a6fc, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=265, earliestPutTs=1681314929284 2023-04-12 15:55:33,134 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:33,134 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 89a61f9dcfec467a921eb25e7ee1f656, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=232, earliestPutTs=1681314928567 2023-04-12 15:55:33,135 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting c9146fc2043c482c9522d42863f16176, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=286, earliestPutTs=1681314929936 2023-04-12 15:55:33,135 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 3e19d4e27579461eada55849c5cb3a3b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=251, earliestPutTs=1681314928617 2023-04-12 15:55:33,135 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 769bc65e5d2e4daa9d03ea643e36dc6e, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=300, earliestPutTs=1681314931510 2023-04-12 15:55:33,136 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 116bd8356ff34924999e86daf7547bbe, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=265, earliestPutTs=1681314929284 2023-04-12 15:55:33,136 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 7a138e876f0247b4a2623c6fd91c8cd8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=286, earliestPutTs=1681314929936 2023-04-12 15:55:33,136 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 3d00aa0b1734468f87b052ec78b67077, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=300, earliestPutTs=1681314931518 2023-04-12 15:55:33,146 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] throttle.PressureAwareThroughputController(145): d7d8c2ad266b62b02ebd2f4b9a678b1a#B#compaction#381 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:55:33,153 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] throttle.PressureAwareThroughputController(145): d7d8c2ad266b62b02ebd2f4b9a678b1a#A#compaction#380 average throughput is 2.19 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:55:33,233 DEBUG [Thread-2432] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:33,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:33,236 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:33,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:33,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:33,237 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:33,237 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:33,237 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:33,237 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:33,237 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:33,237 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:33,239 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:33,239 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:33,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:33,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:33,239 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:33,239 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:33,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:33,239 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:33,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:33,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:33,240 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:33,240 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:33,240 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:33,240 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:33,240 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:33,240 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:33,240 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:33,240 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:33,240 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:33,240 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. started... 2023-04-12 15:55:33,241 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d7d8c2ad266b62b02ebd2f4b9a678b1a 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-12 15:55:33,241 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7d8c2ad266b62b02ebd2f4b9a678b1a, store=A 2023-04-12 15:55:33,241 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:33,241 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7d8c2ad266b62b02ebd2f4b9a678b1a, store=B 2023-04-12 15:55:33,241 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:33,241 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7d8c2ad266b62b02ebd2f4b9a678b1a, store=C 2023-04-12 15:55:33,241 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:33,259 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230412e59baea78e5843e3bca6f7565c90f72f to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412e59baea78e5843e3bca6f7565c90f72f 2023-04-12 15:55:33,270 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=318, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/A/b959bed4014944109878cdc0d49d2513 2023-04-12 15:55:33,561 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/B/f93dc0e3f4a741baa0210b1351b16d45 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/f93dc0e3f4a741baa0210b1351b16d45 2023-04-12 15:55:33,567 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in d7d8c2ad266b62b02ebd2f4b9a678b1a/B of d7d8c2ad266b62b02ebd2f4b9a678b1a into f93dc0e3f4a741baa0210b1351b16d45(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:55:33,567 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d7d8c2ad266b62b02ebd2f4b9a678b1a: 2023-04-12 15:55:33,567 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a., storeName=d7d8c2ad266b62b02ebd2f4b9a678b1a/B, priority=11, startTime=1681314933127; duration=0sec 2023-04-12 15:55:33,567 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:55:33,567 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-12 15:55:33,572 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61176 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-12 15:55:33,572 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1912): d7d8c2ad266b62b02ebd2f4b9a678b1a/C is initiating minor compaction (all files) 2023-04-12 15:55:33,572 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d7d8c2ad266b62b02ebd2f4b9a678b1a/C in TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:33,573 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/5decac26913342389fd06c008c80097a, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/b568c95af0734d8cb036046fc3346665, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/bf8e37a5850d4275a01ca80f8b02939a, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/a0ee708199a44bfc9e33c2bab40ef921, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/ac4fd241b7e24b2c8edf95cc91bd46cd] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp, totalSize=59.7 K 2023-04-12 15:55:33,573 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 5decac26913342389fd06c008c80097a, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=232, earliestPutTs=1681314928567 2023-04-12 15:55:33,573 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting b568c95af0734d8cb036046fc3346665, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=251, earliestPutTs=1681314928617 2023-04-12 15:55:33,574 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting bf8e37a5850d4275a01ca80f8b02939a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=265, earliestPutTs=1681314929284 2023-04-12 15:55:33,574 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting a0ee708199a44bfc9e33c2bab40ef921, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=286, earliestPutTs=1681314929936 2023-04-12 15:55:33,574 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting ac4fd241b7e24b2c8edf95cc91bd46cd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=300, earliestPutTs=1681314931518 2023-04-12 15:55:33,593 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] throttle.PressureAwareThroughputController(145): d7d8c2ad266b62b02ebd2f4b9a678b1a#C#compaction#384 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:55:33,684 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=318 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/B/6a02019efb9940f2b47d98ec4583a962 2023-04-12 15:55:33,983 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/A/ebee2b3b4e7243a6aea7c34b0acd9f3d as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/ebee2b3b4e7243a6aea7c34b0acd9f3d 2023-04-12 15:55:33,986 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in d7d8c2ad266b62b02ebd2f4b9a678b1a/A of d7d8c2ad266b62b02ebd2f4b9a678b1a into ebee2b3b4e7243a6aea7c34b0acd9f3d(size=26.1 K), total size for store is 26.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:55:33,986 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d7d8c2ad266b62b02ebd2f4b9a678b1a: 2023-04-12 15:55:33,986 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a., storeName=d7d8c2ad266b62b02ebd2f4b9a678b1a/A, priority=11, startTime=1681314933127; duration=0sec 2023-04-12 15:55:33,987 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:55:34,001 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/C/9348125373ea4f269cb2df11cb5245fd as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/9348125373ea4f269cb2df11cb5245fd 2023-04-12 15:55:34,005 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in d7d8c2ad266b62b02ebd2f4b9a678b1a/C of d7d8c2ad266b62b02ebd2f4b9a678b1a into 9348125373ea4f269cb2df11cb5245fd(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:55:34,005 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d7d8c2ad266b62b02ebd2f4b9a678b1a: 2023-04-12 15:55:34,005 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a., storeName=d7d8c2ad266b62b02ebd2f4b9a678b1a/C, priority=11, startTime=1681314933128; duration=0sec 2023-04-12 15:55:34,005 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:55:34,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on d7d8c2ad266b62b02ebd2f4b9a678b1a 2023-04-12 15:55:34,085 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. as already flushing 2023-04-12 15:55:34,093 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=318 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/C/cf0b6bd86d5b4920b5aa57b14d245a80 2023-04-12 15:55:34,098 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/A/b959bed4014944109878cdc0d49d2513 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/b959bed4014944109878cdc0d49d2513 2023-04-12 15:55:34,102 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/b959bed4014944109878cdc0d49d2513, entries=150, sequenceid=318, filesize=25.4 K 2023-04-12 15:55:34,103 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/B/6a02019efb9940f2b47d98ec4583a962 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/6a02019efb9940f2b47d98ec4583a962 2023-04-12 15:55:34,107 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/6a02019efb9940f2b47d98ec4583a962, entries=150, sequenceid=318, filesize=11.9 K 2023-04-12 15:55:34,108 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/C/cf0b6bd86d5b4920b5aa57b14d245a80 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/cf0b6bd86d5b4920b5aa57b14d245a80 2023-04-12 15:55:34,112 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/cf0b6bd86d5b4920b5aa57b14d245a80, entries=150, sequenceid=318, filesize=11.9 K 2023-04-12 15:55:34,112 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=53.67 KB/54960 for d7d8c2ad266b62b02ebd2f4b9a678b1a in 871ms, sequenceid=318, compaction requested=false 2023-04-12 15:55:34,112 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d7d8c2ad266b62b02ebd2f4b9a678b1a: 2023-04-12 15:55:34,113 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:34,113 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:34,113 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:34,113 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,113 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:34,113 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:34,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on d7d8c2ad266b62b02ebd2f4b9a678b1a 2023-04-12 15:55:34,114 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d7d8c2ad266b62b02ebd2f4b9a678b1a 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-12 15:55:34,114 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:34,115 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7d8c2ad266b62b02ebd2f4b9a678b1a, store=A 2023-04-12 15:55:34,115 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:34,115 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7d8c2ad266b62b02ebd2f4b9a678b1a, store=B 2023-04-12 15:55:34,115 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:34,115 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7d8c2ad266b62b02ebd2f4b9a678b1a, store=C 2023-04-12 15:55:34,115 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:34,115 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:34,115 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:34,116 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:34,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:34,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:34,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:34,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:34,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:34,121 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:34,121 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4981a7d3[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:34,121 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:34,121 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:34,123 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:34,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:34,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:34,123 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:34,123 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:34,123 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:34,123 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,123 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:34,125 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:34,125 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:34,125 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:34,126 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:34,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:34,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:34,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:34,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:34,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:34,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:34,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:34,129 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:34,129 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:34,130 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:34,130 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:34,130 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:34,130 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:34,131 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:34,131 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:34,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:34,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:34,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:34,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:34,131 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:34,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:34,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:34,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:34,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:34,132 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:34,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:34,134 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304123ac86ad5277443838db3c23ed50538fe to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304123ac86ad5277443838db3c23ed50538fe 2023-04-12 15:55:34,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:34,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:34,137 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:34,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:34,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,142 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,142 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:34,142 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:34,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:34,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:34,142 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:34,142 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:34,142 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,142 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:34,142 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:34,142 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:34,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:34,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:34,142 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:34,143 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:34,143 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:34,143 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:34,144 DEBUG [Thread-2432] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:34,144 DEBUG [Thread-2432] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:34,149 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:34,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59810 deadline: 1681314994142, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,150 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:34,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59804 deadline: 1681314994144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,151 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:34,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59902 deadline: 1681314994145, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,151 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:34,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59882 deadline: 1681314994146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,244 DEBUG [Thread-2432] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:34,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:34,247 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:34,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:34,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:34,250 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:34,250 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:34,250 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:34,250 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:34,250 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:34,250 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:34,251 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:34,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:34,252 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:34,252 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,252 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:34,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:34,253 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:34,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:34,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:34,253 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:34,253 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:34,253 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:34,253 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:34,253 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:34,253 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:34,254 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:34,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59810 deadline: 1681314994250, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,254 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:34,254 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:34,254 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:34,254 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:34,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59804 deadline: 1681314994251, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,255 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. started... 2023-04-12 15:55:34,255 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. as already flushing 2023-04-12 15:55:34,255 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:34,255 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:34,255 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:34,255 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:34,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59902 deadline: 1681314994252, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,255 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,255 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:34,255 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:34,255 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:34,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59882 deadline: 1681314994253, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,257 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:34,257 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:34,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:34,257 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:34,257 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:34,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:34,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:34,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:34,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:34,259 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:34,259 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4418eb23[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:34,259 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:34,259 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:34,260 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:34,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:34,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:34,260 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:34,260 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:34,260 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:34,260 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,261 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:34,263 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:34,263 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:34,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:34,263 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:34,263 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:34,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:34,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:34,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:34,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:34,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:34,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:34,266 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:34,266 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:34,266 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:34,266 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:34,266 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:34,266 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:34,269 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:34,269 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:34,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:34,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:34,269 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:34,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:34,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:34,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:34,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:34,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:34,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:34,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:34,270 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:34,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:34,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:34,271 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:34,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:34,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,274 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:34,274 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:34,274 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:34,274 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:34,274 DEBUG [Thread-2432] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:34,274 DEBUG [Thread-2432] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:34,274 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,274 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:34,274 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:34,274 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,274 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:34,274 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:34,275 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:34,275 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:34,275 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:34,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:34,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:34,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:34,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:34,374 DEBUG [Thread-2432] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:34,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:34,376 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:34,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:34,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:34,377 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:34,378 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:34,378 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:34,378 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:34,378 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:34,378 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:34,379 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:34,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:34,379 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:34,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:34,380 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,380 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:34,380 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:34,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:34,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:34,380 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:34,381 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:34,381 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:34,381 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:34,381 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:34,381 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:34,381 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:34,381 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:34,381 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:34,381 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. started... 2023-04-12 15:55:34,381 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. as already flushing 2023-04-12 15:55:34,381 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:34,381 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:34,381 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:34,381 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,381 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:34,381 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:34,384 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,384 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:34,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:34,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:34,384 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:34,385 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:34,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:34,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:34,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:34,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:34,386 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:34,386 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50207b04[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:34,386 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:34,386 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:34,387 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:34,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:34,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:34,387 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:34,387 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:34,387 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:34,387 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,387 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:34,388 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:34,388 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:34,388 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,388 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:34,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:34,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:34,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:34,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:34,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:34,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:34,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:34,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:34,390 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:34,391 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:34,391 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:34,391 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:34,391 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:34,391 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:34,392 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:34,392 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:34,392 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:34,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:34,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:34,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:34,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:34,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:34,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:34,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:34,393 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:34,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:34,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:34,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:34,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:34,394 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:34,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:34,398 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,398 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:34,398 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:34,398 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,398 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:34,398 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:34,398 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:34,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:34,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:34,398 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:34,398 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:34,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:34,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:34,398 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:34,399 DEBUG [Thread-2432] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:34,399 DEBUG [Thread-2432] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:34,399 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:34,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:34,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:34,459 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:34,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59810 deadline: 1681314994455, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,460 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:34,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59804 deadline: 1681314994456, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,460 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:34,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59882 deadline: 1681314994456, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,462 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:34,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59902 deadline: 1681314994457, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,499 DEBUG [Thread-2432] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:34,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:34,501 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:34,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:34,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:34,503 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:34,503 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:34,503 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:34,504 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:34,504 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:34,504 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:34,505 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,505 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:34,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:34,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:34,506 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,506 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:34,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:34,506 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:34,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:34,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:34,507 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:34,507 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:34,507 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:34,507 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:34,507 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:34,507 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:34,507 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:34,507 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:34,507 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:34,507 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. started... 2023-04-12 15:55:34,508 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. as already flushing 2023-04-12 15:55:34,508 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:34,508 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:34,508 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:34,508 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,508 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:34,508 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:34,509 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,509 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:34,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:34,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:34,510 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:34,510 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:34,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:34,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:34,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:34,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:34,512 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:34,512 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f21662[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:34,512 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:34,513 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:34,514 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:34,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:34,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:34,514 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:34,514 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:34,514 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:34,514 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,514 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:34,516 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:34,517 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:34,516 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,517 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:34,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:34,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:34,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:34,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:34,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:34,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:34,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:34,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:34,519 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:34,519 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:34,520 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:34,520 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:34,520 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:34,520 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:34,521 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:34,521 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:34,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:34,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:34,522 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:34,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:34,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:34,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:34,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:34,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:34,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:34,522 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:34,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:34,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:34,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:34,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,525 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:34,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:34,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,530 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,530 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:34,530 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:34,530 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:34,530 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:34,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:34,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:34,530 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:34,530 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:34,530 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,530 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:34,531 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:34,531 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:34,530 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:34,531 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:34,531 DEBUG [Thread-2432] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:34,531 DEBUG [Thread-2432] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:34,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:34,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:34,539 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=333, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/A/60eaba89351f477f81c0852406aaee26 2023-04-12 15:55:34,631 DEBUG [Thread-2432] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:34,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:34,633 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:34,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:34,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:34,634 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:34,634 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:34,634 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:34,634 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:34,634 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:34,634 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:34,636 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:34,636 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:34,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:34,637 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,637 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:34,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:34,637 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:34,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:34,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:34,638 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:34,638 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:34,638 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:34,638 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:34,638 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:34,638 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:34,638 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:34,638 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:34,638 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:34,638 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. started... 2023-04-12 15:55:34,638 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. as already flushing 2023-04-12 15:55:34,638 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:34,638 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:34,638 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:34,638 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,639 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:34,639 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:34,640 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,640 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:34,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:34,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:34,640 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:34,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:34,640 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:34,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:34,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:34,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:34,642 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:34,642 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@460986d0[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:34,642 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:34,642 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:34,643 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:34,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:34,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:34,643 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,643 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:34,643 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:34,643 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:34,643 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:34,645 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:34,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:34,648 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:34,648 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:34,648 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:34,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:34,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:34,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:34,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:34,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:34,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:34,658 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:34,659 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:34,659 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:34,659 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:34,659 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:34,659 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:34,662 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:34,662 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:34,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:34,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:34,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:34,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:34,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:34,662 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:34,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:34,663 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:34,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:34,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:34,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:34,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:34,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:34,664 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:34,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:34,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,667 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:34,667 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:34,667 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:34,667 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:34,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:34,667 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:34,667 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:34,667 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:34,667 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:34,667 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:34,667 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,667 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:34,667 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:34,667 DEBUG [Thread-2432] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:34,667 DEBUG [Thread-2432] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:34,667 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:34,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:34,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:34,764 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:34,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59810 deadline: 1681314994763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,768 DEBUG [Thread-2432] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:34,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:34,769 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:34,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59902 deadline: 1681314994764, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,769 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:34,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59882 deadline: 1681314994768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,770 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:34,770 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:34,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59804 deadline: 1681314994768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:34,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:34,776 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:34,776 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:34,777 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:34,777 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:34,777 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:34,777 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:34,779 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:34,779 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:34,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:34,779 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,779 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:34,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:34,779 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:34,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:34,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:34,780 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:34,780 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:34,780 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:34,780 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:34,780 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:34,780 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:34,781 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:34,781 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:34,781 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:34,781 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. started... 2023-04-12 15:55:34,781 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. as already flushing 2023-04-12 15:55:34,781 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:34,781 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:34,781 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:34,781 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,781 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:34,781 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:34,784 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:34,784 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:34,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:34,784 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:34,784 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:34,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:34,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:34,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:34,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:34,786 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:34,786 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45cb2cc1[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:34,786 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:34,786 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:34,788 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:34,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:34,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:34,788 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:34,788 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:34,788 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:34,788 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,788 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:34,790 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:34,790 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:34,790 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:34,790 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:34,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:34,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:34,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:34,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:34,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:34,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:34,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:34,793 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:34,793 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:34,793 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:34,793 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:34,793 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:34,793 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:34,797 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:34,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:34,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:34,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:34,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:34,798 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:34,798 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:34,798 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:34,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:34,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:34,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:34,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:34,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:34,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:34,799 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:34,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:34,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:34,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,801 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,802 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:34,802 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:34,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:34,802 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:34,802 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:34,802 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:34,802 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:34,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:34,802 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:34,802 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,802 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:34,802 DEBUG [Thread-2432] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:34,802 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:34,802 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:34,802 DEBUG [Thread-2432] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:34,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:34,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:34,902 DEBUG [Thread-2432] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:34,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:34,906 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:34,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:34,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:34,907 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:34,907 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:34,907 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:34,907 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:34,907 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:34,907 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:34,909 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:34,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:34,910 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:34,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:34,911 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:34,911 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,911 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:34,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:34,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:34,911 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:34,911 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:34,911 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:34,911 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:34,911 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:34,912 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:34,912 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:34,912 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:34,912 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. started... 2023-04-12 15:55:34,912 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. as already flushing 2023-04-12 15:55:34,912 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:34,912 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:34,912 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:34,912 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,912 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:34,912 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:34,914 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:34,914 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:34,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:34,914 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:34,914 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:34,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:34,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:34,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:34,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:34,917 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:34,917 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16e08727[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:34,917 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:34,917 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:34,920 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:34,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:34,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:34,921 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:34,921 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:34,921 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:34,921 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,921 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:34,922 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,922 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:34,922 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:34,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:34,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:34,923 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:34,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:34,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:34,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:34,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:34,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:34,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:34,926 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:34,926 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:34,926 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:34,926 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:34,926 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:34,926 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:34,929 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:34,929 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:34,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:34,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:34,929 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:34,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:34,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:34,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:34,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:34,930 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:34,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:34,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:34,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:34,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:34,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:34,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,932 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:34,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:34,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,936 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:34,936 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:34,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:34,936 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:34,936 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:34,936 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:34,936 DEBUG [Thread-2432] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:34,936 DEBUG [Thread-2432] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:34,936 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:34,936 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:34,936 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:34,936 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:34,936 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:34,937 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:34,937 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:34,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:34,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:34,950 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=333 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/B/755dc88ee6f142689159e9bebf7bc43a 2023-04-12 15:55:35,036 DEBUG [Thread-2432] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:35,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:35,040 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:35,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:35,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:35,041 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:35,041 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:35,041 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:35,041 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:35,042 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:35,042 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:35,043 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:35,043 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:35,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:35,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:35,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:35,044 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:35,044 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:35,044 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:35,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:35,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:35,045 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:35,046 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:35,046 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:35,046 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:35,046 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:35,046 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:35,048 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:35,048 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:35,048 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:35,048 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. started... 2023-04-12 15:55:35,048 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. as already flushing 2023-04-12 15:55:35,048 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:35,048 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:35,048 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:35,048 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:35,048 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:35,048 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:35,054 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:35,055 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:35,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:35,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:35,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:35,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:35,056 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:35,056 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:35,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:35,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:35,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:35,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:35,060 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:35,060 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b3d29f[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:35,060 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:35,060 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:35,061 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:35,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:35,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:35,062 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:35,062 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:35,062 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:35,062 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:35,062 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:35,063 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:35,063 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:35,063 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:35,064 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:35,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:35,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:35,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:35,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:35,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:35,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:35,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:35,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:35,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:35,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:35,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:35,066 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:35,066 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:35,066 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:35,066 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:35,066 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:35,066 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:35,068 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:35,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:35,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:35,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:35,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:35,068 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:35,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:35,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:35,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:35,069 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:35,069 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:35,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:35,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:35,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:35,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:35,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:35,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:35,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:35,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:35,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:35,073 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:35,076 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:35,076 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:35,076 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:35,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:35,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:35,076 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:35,076 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:35,076 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:35,076 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:35,076 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:35,077 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:35,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:35,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:35,076 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:35,077 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:35,077 DEBUG [Thread-2432] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:35,077 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:35,077 DEBUG [Thread-2432] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:35,177 DEBUG [Thread-2432] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:35,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:35,184 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:35,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:35,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:35,185 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:35,185 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:35,185 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:35,185 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:35,185 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:35,185 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:35,187 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:35,187 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:35,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:35,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:35,187 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:35,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:35,187 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:35,188 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:35,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:35,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:35,189 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:35,190 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:35,190 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:35,190 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:35,190 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:35,190 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:35,192 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:35,192 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:35,192 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:35,192 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. started... 2023-04-12 15:55:35,192 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. as already flushing 2023-04-12 15:55:35,192 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:35,192 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:35,192 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:35,192 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:35,192 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:35,192 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:35,194 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:35,194 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:35,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:35,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:35,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:35,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:35,194 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:35,194 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:35,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:35,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:35,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:35,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:35,196 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:35,196 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e06fbdf[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:35,196 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:35,196 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:35,198 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:35,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:35,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:35,198 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:35,198 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:35,198 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:35,198 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:35,198 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:35,200 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:35,200 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:35,200 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:35,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:35,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:35,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:35,201 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:35,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:35,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:35,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:35,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:35,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:35,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:35,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:35,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:35,204 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:35,204 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:35,204 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:35,204 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:35,204 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:35,204 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:35,206 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:35,206 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:35,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:35,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:35,206 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:35,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:35,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:35,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:35,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:35,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:35,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:35,207 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:35,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:35,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:35,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:35,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:35,210 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:35,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:35,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:35,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:35,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:35,214 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:35,214 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:35,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:35,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:35,214 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:35,214 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:35,214 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:35,214 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:35,214 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:35,214 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:35,215 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:35,215 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:35,214 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:35,215 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:35,215 DEBUG [Thread-2432] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:35,215 DEBUG [Thread-2432] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:35,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:35,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:35,270 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:35,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59810 deadline: 1681314995266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:35,272 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:35,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59902 deadline: 1681314995271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:35,276 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:35,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59804 deadline: 1681314995272, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:35,278 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:35,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59882 deadline: 1681314995273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:35,315 DEBUG [Thread-2432] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:35,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:35,317 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:35,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:35,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:35,318 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:35,318 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:35,319 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:35,319 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:35,319 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:35,319 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:35,321 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:35,321 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:35,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:35,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:35,321 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:35,321 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:35,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:35,322 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:35,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:35,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:35,322 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:35,322 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:35,322 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:35,322 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:35,323 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:35,323 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:35,323 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:35,323 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:35,323 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:35,323 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. started... 2023-04-12 15:55:35,323 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. as already flushing 2023-04-12 15:55:35,323 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:35,323 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:35,323 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:35,323 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:35,323 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:35,323 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:35,325 DEBUG [Thread-2435] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1ec12b02 to 127.0.0.1:60154 2023-04-12 15:55:35,325 DEBUG [Thread-2435] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 15:55:35,325 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:35,325 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:35,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:35,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:35,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:35,326 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:35,326 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:35,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:35,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:35,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:35,327 DEBUG [Thread-2441] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x159c5e0c to 127.0.0.1:60154 2023-04-12 15:55:35,327 DEBUG [Thread-2441] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 15:55:35,327 DEBUG [Thread-2437] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1d7ba242 to 127.0.0.1:60154 2023-04-12 15:55:35,328 DEBUG [Thread-2437] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 15:55:35,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:35,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:35,330 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:35,330 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12b773fa[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:35,330 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:35,330 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:35,332 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:35,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:35,332 DEBUG [Thread-2439] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3e079da1 to 127.0.0.1:60154 2023-04-12 15:55:35,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:35,332 DEBUG [Thread-2439] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 15:55:35,332 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:35,332 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:35,332 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:35,332 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:35,332 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:35,333 DEBUG [Thread-2433] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x68a0e21a to 127.0.0.1:60154 2023-04-12 15:55:35,333 DEBUG [Thread-2433] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 15:55:35,335 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:35,335 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:35,335 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:35,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:35,335 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:35,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:35,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:35,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:35,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:35,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:35,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:35,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:35,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:35,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:35,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:35,337 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:35,337 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:35,337 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:35,337 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:35,337 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:35,337 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:35,339 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:35,339 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:35,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:35,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:35,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:35,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:35,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:35,339 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:35,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:35,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:35,339 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:35,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:35,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:35,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:35,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:35,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:35,340 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:35,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:35,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:35,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:35,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:35,343 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:35,344 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:35,344 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:35,344 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:35,344 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:35,344 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:35,344 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:35,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:35,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:35,344 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:35,344 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:35,344 DEBUG [Thread-2432] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:35,344 DEBUG [Thread-2432] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:35,344 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:35,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:35,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:35,344 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:35,344 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:35,374 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=333 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/C/5979e53a22e544dc98359ee996f7b887 2023-04-12 15:55:35,377 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/A/60eaba89351f477f81c0852406aaee26 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/60eaba89351f477f81c0852406aaee26 2023-04-12 15:55:35,380 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/60eaba89351f477f81c0852406aaee26, entries=200, sequenceid=333, filesize=32.3 K 2023-04-12 15:55:35,381 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/B/755dc88ee6f142689159e9bebf7bc43a as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/755dc88ee6f142689159e9bebf7bc43a 2023-04-12 15:55:35,384 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/755dc88ee6f142689159e9bebf7bc43a, entries=150, sequenceid=333, filesize=11.9 K 2023-04-12 15:55:35,384 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/C/5979e53a22e544dc98359ee996f7b887 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/5979e53a22e544dc98359ee996f7b887 2023-04-12 15:55:35,387 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/5979e53a22e544dc98359ee996f7b887, entries=150, sequenceid=333, filesize=11.9 K 2023-04-12 15:55:35,388 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for d7d8c2ad266b62b02ebd2f4b9a678b1a in 1274ms, sequenceid=333, compaction requested=true 2023-04-12 15:55:35,388 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d7d8c2ad266b62b02ebd2f4b9a678b1a: 2023-04-12 15:55:35,388 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:55:35,388 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:55:35,388 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 15:55:35,388 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 15:55:35,388 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:55:35,389 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85894 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 15:55:35,389 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1912): d7d8c2ad266b62b02ebd2f4b9a678b1a/A is initiating minor compaction (all files) 2023-04-12 15:55:35,389 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37274 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 15:55:35,389 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d7d8c2ad266b62b02ebd2f4b9a678b1a/A in TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:35,389 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1912): d7d8c2ad266b62b02ebd2f4b9a678b1a/B is initiating minor compaction (all files) 2023-04-12 15:55:35,389 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/ebee2b3b4e7243a6aea7c34b0acd9f3d, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/b959bed4014944109878cdc0d49d2513, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/60eaba89351f477f81c0852406aaee26] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp, totalSize=83.9 K 2023-04-12 15:55:35,389 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d7d8c2ad266b62b02ebd2f4b9a678b1a/B in TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:35,389 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/f93dc0e3f4a741baa0210b1351b16d45, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/6a02019efb9940f2b47d98ec4583a962, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/755dc88ee6f142689159e9bebf7bc43a] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp, totalSize=36.4 K 2023-04-12 15:55:35,389 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting ebee2b3b4e7243a6aea7c34b0acd9f3d, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=300, earliestPutTs=1681314931518 2023-04-12 15:55:35,389 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting f93dc0e3f4a741baa0210b1351b16d45, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=300, earliestPutTs=1681314931518 2023-04-12 15:55:35,390 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting b959bed4014944109878cdc0d49d2513, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=318, earliestPutTs=1681314931915 2023-04-12 15:55:35,390 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 6a02019efb9940f2b47d98ec4583a962, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=318, earliestPutTs=1681314931915 2023-04-12 15:55:35,390 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 60eaba89351f477f81c0852406aaee26, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=333, earliestPutTs=1681314934101 2023-04-12 15:55:35,390 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 755dc88ee6f142689159e9bebf7bc43a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=333, earliestPutTs=1681314934108 2023-04-12 15:55:35,395 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] throttle.PressureAwareThroughputController(145): d7d8c2ad266b62b02ebd2f4b9a678b1a#B#compaction#389 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:55:35,397 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] throttle.PressureAwareThroughputController(145): d7d8c2ad266b62b02ebd2f4b9a678b1a#A#compaction#390 average throughput is 6.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:55:35,444 DEBUG [Thread-2432] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:35,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:35,802 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/B/9ef6920c23fb430cae09b4853b86ac42 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/9ef6920c23fb430cae09b4853b86ac42 2023-04-12 15:55:35,806 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d7d8c2ad266b62b02ebd2f4b9a678b1a/B of d7d8c2ad266b62b02ebd2f4b9a678b1a into 9ef6920c23fb430cae09b4853b86ac42(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:55:35,806 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d7d8c2ad266b62b02ebd2f4b9a678b1a: 2023-04-12 15:55:35,806 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a., storeName=d7d8c2ad266b62b02ebd2f4b9a678b1a/B, priority=13, startTime=1681314935388; duration=0sec 2023-04-12 15:55:35,806 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:55:35,806 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 15:55:35,807 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37274 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 15:55:35,807 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1912): d7d8c2ad266b62b02ebd2f4b9a678b1a/C is initiating minor compaction (all files) 2023-04-12 15:55:35,807 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d7d8c2ad266b62b02ebd2f4b9a678b1a/C in TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:35,808 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/9348125373ea4f269cb2df11cb5245fd, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/cf0b6bd86d5b4920b5aa57b14d245a80, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/5979e53a22e544dc98359ee996f7b887] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp, totalSize=36.4 K 2023-04-12 15:55:35,808 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 9348125373ea4f269cb2df11cb5245fd, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=300, earliestPutTs=1681314931518 2023-04-12 15:55:35,808 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting cf0b6bd86d5b4920b5aa57b14d245a80, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=318, earliestPutTs=1681314931915 2023-04-12 15:55:35,808 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 5979e53a22e544dc98359ee996f7b887, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=333, earliestPutTs=1681314934108 2023-04-12 15:55:35,813 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] throttle.PressureAwareThroughputController(145): d7d8c2ad266b62b02ebd2f4b9a678b1a#C#compaction#391 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:55:36,213 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/A/c6ad090eeea84bc3bc21ee14e48cd9f1 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/c6ad090eeea84bc3bc21ee14e48cd9f1 2023-04-12 15:55:36,217 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d7d8c2ad266b62b02ebd2f4b9a678b1a/A of d7d8c2ad266b62b02ebd2f4b9a678b1a into c6ad090eeea84bc3bc21ee14e48cd9f1(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:55:36,218 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d7d8c2ad266b62b02ebd2f4b9a678b1a: 2023-04-12 15:55:36,218 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a., storeName=d7d8c2ad266b62b02ebd2f4b9a678b1a/A, priority=13, startTime=1681314935388; duration=0sec 2023-04-12 15:55:36,218 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:55:36,222 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/C/d86c6c8ae0254fecb11ecdfb21a06410 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/d86c6c8ae0254fecb11ecdfb21a06410 2023-04-12 15:55:36,226 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d7d8c2ad266b62b02ebd2f4b9a678b1a/C of d7d8c2ad266b62b02ebd2f4b9a678b1a into d86c6c8ae0254fecb11ecdfb21a06410(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:55:36,226 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d7d8c2ad266b62b02ebd2f4b9a678b1a: 2023-04-12 15:55:36,226 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a., storeName=d7d8c2ad266b62b02ebd2f4b9a678b1a/C, priority=13, startTime=1681314935388; duration=0sec 2023-04-12 15:55:36,226 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:55:36,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on d7d8c2ad266b62b02ebd2f4b9a678b1a 2023-04-12 15:55:36,273 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d7d8c2ad266b62b02ebd2f4b9a678b1a 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-12 15:55:36,273 DEBUG [Thread-2428] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4fe1db6a to 127.0.0.1:60154 2023-04-12 15:55:36,273 DEBUG [Thread-2428] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 15:55:36,273 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7d8c2ad266b62b02ebd2f4b9a678b1a, store=A 2023-04-12 15:55:36,273 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:55:36,273 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7d8c2ad266b62b02ebd2f4b9a678b1a, store=B 2023-04-12 15:55:36,273 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:55:36,273 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7d8c2ad266b62b02ebd2f4b9a678b1a, store=C 2023-04-12 15:55:36,273 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:55:36,283 DEBUG [Thread-2430] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x10377b50 to 127.0.0.1:60154 2023-04-12 15:55:36,283 DEBUG [Thread-2430] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 15:55:36,285 DEBUG [Thread-2426] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3620ee35 to 127.0.0.1:60154 2023-04-12 15:55:36,285 DEBUG [Thread-2426] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 15:55:36,286 DEBUG [Thread-2424] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x71b5058b to 127.0.0.1:60154 2023-04-12 15:55:36,287 DEBUG [Thread-2424] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 15:55:36,684 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230412037f90c7c1e14432ae213b8660263de4 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412037f90c7c1e14432ae213b8660263de4 2023-04-12 15:55:36,691 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=357, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/A/86c48de155cc4823805c76ad8a416243 2023-04-12 15:55:36,700 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=357 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/B/1816f0145e2447c2bc08bf2815e16414 2023-04-12 15:55:37,112 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=357 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/C/dc8f3403d05a4df1be2974bec6187c2f 2023-04-12 15:55:37,116 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/A/86c48de155cc4823805c76ad8a416243 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/86c48de155cc4823805c76ad8a416243 2023-04-12 15:55:37,120 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/86c48de155cc4823805c76ad8a416243, entries=150, sequenceid=357, filesize=25.4 K 2023-04-12 15:55:37,121 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/B/1816f0145e2447c2bc08bf2815e16414 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/1816f0145e2447c2bc08bf2815e16414 2023-04-12 15:55:37,125 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/1816f0145e2447c2bc08bf2815e16414, entries=150, sequenceid=357, filesize=11.9 K 2023-04-12 15:55:37,126 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/C/dc8f3403d05a4df1be2974bec6187c2f as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/dc8f3403d05a4df1be2974bec6187c2f 2023-04-12 15:55:37,130 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/dc8f3403d05a4df1be2974bec6187c2f, entries=150, sequenceid=357, filesize=11.9 K 2023-04-12 15:55:37,130 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=20.13 KB/20610 for d7d8c2ad266b62b02ebd2f4b9a678b1a in 857ms, sequenceid=357, compaction requested=false 2023-04-12 15:55:37,130 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d7d8c2ad266b62b02ebd2f4b9a678b1a: 2023-04-12 15:55:37,836 DEBUG [Thread-2422] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x38341134 to 127.0.0.1:60154 2023-04-12 15:55:37,836 DEBUG [Thread-2422] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 15:55:37,836 INFO [Listener at localhost/43583] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-12 15:55:37,836 INFO [Listener at localhost/43583] hbase.AcidGuaranteesTestTool(394): wrote 22 2023-04-12 15:55:37,836 INFO [Listener at localhost/43583] hbase.AcidGuaranteesTestTool(394): wrote 64 2023-04-12 15:55:37,836 INFO [Listener at localhost/43583] hbase.AcidGuaranteesTestTool(394): wrote 51 2023-04-12 15:55:37,837 INFO [Listener at localhost/43583] hbase.AcidGuaranteesTestTool(394): wrote 40 2023-04-12 15:55:37,837 INFO [Listener at localhost/43583] hbase.AcidGuaranteesTestTool(394): wrote 102 2023-04-12 15:55:37,837 INFO [Listener at localhost/43583] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-12 15:55:37,837 INFO [Listener at localhost/43583] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-12 15:55:37,837 INFO [Listener at localhost/43583] hbase.AcidGuaranteesTestTool(402): scanned 1829 2023-04-12 15:55:37,837 INFO [Listener at localhost/43583] hbase.AcidGuaranteesTestTool(403): verified 5486 rows 2023-04-12 15:55:37,837 INFO [Listener at localhost/43583] hbase.AcidGuaranteesTestTool(402): scanned 1825 2023-04-12 15:55:37,837 INFO [Listener at localhost/43583] hbase.AcidGuaranteesTestTool(403): verified 5475 rows 2023-04-12 15:55:37,837 INFO [Listener at localhost/43583] hbase.AcidGuaranteesTestTool(402): scanned 1796 2023-04-12 15:55:37,837 INFO [Listener at localhost/43583] hbase.AcidGuaranteesTestTool(403): verified 5388 rows 2023-04-12 15:55:37,837 INFO [Listener at localhost/43583] hbase.AcidGuaranteesTestTool(402): scanned 1802 2023-04-12 15:55:37,837 INFO [Listener at localhost/43583] hbase.AcidGuaranteesTestTool(403): verified 5406 rows 2023-04-12 15:55:37,837 INFO [Listener at localhost/43583] hbase.AcidGuaranteesTestTool(402): scanned 1816 2023-04-12 15:55:37,837 INFO [Listener at localhost/43583] hbase.AcidGuaranteesTestTool(403): verified 5448 rows 2023-04-12 15:55:37,837 INFO [Listener at localhost/43583] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-12 15:55:37,837 DEBUG [Listener at localhost/43583] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0bcfc1e0 to 127.0.0.1:60154 2023-04-12 15:55:37,837 DEBUG [Listener at localhost/43583] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 15:55:37,839 INFO [Listener at localhost/43583] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-12 15:55:37,839 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-12 15:55:37,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure2.ProcedureExecutor(1029): Stored pid=43, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-12 15:55:37,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-12 15:55:37,844 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681314937844"}]},"ts":"1681314937844"} 2023-04-12 15:55:37,845 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-12 15:55:37,847 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-12 15:55:37,847 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=d7d8c2ad266b62b02ebd2f4b9a678b1a, UNASSIGN}] 2023-04-12 15:55:37,849 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=d7d8c2ad266b62b02ebd2f4b9a678b1a, UNASSIGN 2023-04-12 15:55:37,849 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=d7d8c2ad266b62b02ebd2f4b9a678b1a, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:37,849 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681314937849"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681314937849"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681314937849"}]},"ts":"1681314937849"} 2023-04-12 15:55:37,850 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=45, ppid=44, state=RUNNABLE; CloseRegionProcedure d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739}] 2023-04-12 15:55:37,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-12 15:55:38,002 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close d7d8c2ad266b62b02ebd2f4b9a678b1a 2023-04-12 15:55:38,002 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing d7d8c2ad266b62b02ebd2f4b9a678b1a, disabling compactions & flushes 2023-04-12 15:55:38,002 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:38,002 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:38,002 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. after waiting 0 ms 2023-04-12 15:55:38,002 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:38,002 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing d7d8c2ad266b62b02ebd2f4b9a678b1a 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-12 15:55:38,002 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7d8c2ad266b62b02ebd2f4b9a678b1a, store=A 2023-04-12 15:55:38,002 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:38,002 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7d8c2ad266b62b02ebd2f4b9a678b1a, store=B 2023-04-12 15:55:38,002 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:38,002 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7d8c2ad266b62b02ebd2f4b9a678b1a, store=C 2023-04-12 15:55:38,002 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:38,012 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230412e9d885533bdc440fa234e7f0948fcd7e to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412e9d885533bdc440fa234e7f0948fcd7e 2023-04-12 15:55:38,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-12 15:55:38,420 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=364, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/A/f6a5a5937d1b4231ace07f50cfb30422 2023-04-12 15:55:38,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-12 15:55:38,828 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=364 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/B/ae0114a0023b4748b35404bd84f23620 2023-04-12 15:55:38,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-12 15:55:39,237 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=364 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/C/e1c5d6f99fc94fbdb0d8c1512e64ef22 2023-04-12 15:55:39,241 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/A/f6a5a5937d1b4231ace07f50cfb30422 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/f6a5a5937d1b4231ace07f50cfb30422 2023-04-12 15:55:39,248 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/f6a5a5937d1b4231ace07f50cfb30422, entries=100, sequenceid=364, filesize=18.6 K 2023-04-12 15:55:39,248 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/B/ae0114a0023b4748b35404bd84f23620 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/ae0114a0023b4748b35404bd84f23620 2023-04-12 15:55:39,252 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/ae0114a0023b4748b35404bd84f23620, entries=100, sequenceid=364, filesize=9.5 K 2023-04-12 15:55:39,252 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/.tmp/C/e1c5d6f99fc94fbdb0d8c1512e64ef22 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/e1c5d6f99fc94fbdb0d8c1512e64ef22 2023-04-12 15:55:39,255 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/e1c5d6f99fc94fbdb0d8c1512e64ef22, entries=100, sequenceid=364, filesize=9.5 K 2023-04-12 15:55:39,256 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for d7d8c2ad266b62b02ebd2f4b9a678b1a in 1254ms, sequenceid=364, compaction requested=true 2023-04-12 15:55:39,285 DEBUG [StoreCloser-TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/3d3c11c962474d92b3abc65caa0fcffe, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/30e2dd705f1a4a58a737ef2d51ce87b8, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/a3bdbb5652f0492981ae9d35148ee94e, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/a1365fc1f5df41c19d4e0a660c14d627, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/2b512416a6da4c3c8c4aee846f2ca3d7, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/62afe8cc6dcd4d8ba4716ed8b68b9f05, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/2fc54e85e7564141ac2214ab1e687675, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/51f72dfe3a574383a083a77dd4aa729f, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/d26fc0e341114b6ab266042e7676a475, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/d3e5c62918d04cada2d1a42c03a26711, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/4a04250de0db44d7bf907caeebc7d05c, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/2406a5d42e7744ef8a79db3374b20945, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/bd87a964601b4b7caa7f139b5fb0af53, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/6a67a5c9dc714fb8bae990f7bc098db7, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/2484a203716e4f4b96e33c5050ea7aa1, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/bb1d6b402f6646aaaad1bde2a3eb29ea, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/da4fce448dce450987a91a7ed8dd7054, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/fe3f3d0e2bc24fa283df4bf8230c13af, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/50714ded97514e99be8da3c958b8a6fc, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/c9146fc2043c482c9522d42863f16176, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/769bc65e5d2e4daa9d03ea643e36dc6e, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/ebee2b3b4e7243a6aea7c34b0acd9f3d, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/b959bed4014944109878cdc0d49d2513, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/60eaba89351f477f81c0852406aaee26] to archive 2023-04-12 15:55:39,286 DEBUG [StoreCloser-TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-12 15:55:39,288 DEBUG [StoreCloser-TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/3d3c11c962474d92b3abc65caa0fcffe to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/3d3c11c962474d92b3abc65caa0fcffe 2023-04-12 15:55:39,289 DEBUG [StoreCloser-TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/30e2dd705f1a4a58a737ef2d51ce87b8 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/30e2dd705f1a4a58a737ef2d51ce87b8 2023-04-12 15:55:39,290 DEBUG [StoreCloser-TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/a3bdbb5652f0492981ae9d35148ee94e to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/a3bdbb5652f0492981ae9d35148ee94e 2023-04-12 15:55:39,291 DEBUG [StoreCloser-TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/a1365fc1f5df41c19d4e0a660c14d627 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/a1365fc1f5df41c19d4e0a660c14d627 2023-04-12 15:55:39,292 DEBUG [StoreCloser-TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/2b512416a6da4c3c8c4aee846f2ca3d7 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/2b512416a6da4c3c8c4aee846f2ca3d7 2023-04-12 15:55:39,293 DEBUG [StoreCloser-TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/62afe8cc6dcd4d8ba4716ed8b68b9f05 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/62afe8cc6dcd4d8ba4716ed8b68b9f05 2023-04-12 15:55:39,294 DEBUG [StoreCloser-TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/2fc54e85e7564141ac2214ab1e687675 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/2fc54e85e7564141ac2214ab1e687675 2023-04-12 15:55:39,295 DEBUG [StoreCloser-TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/51f72dfe3a574383a083a77dd4aa729f to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/51f72dfe3a574383a083a77dd4aa729f 2023-04-12 15:55:39,296 DEBUG [StoreCloser-TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/d26fc0e341114b6ab266042e7676a475 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/d26fc0e341114b6ab266042e7676a475 2023-04-12 15:55:39,296 DEBUG [StoreCloser-TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/d3e5c62918d04cada2d1a42c03a26711 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/d3e5c62918d04cada2d1a42c03a26711 2023-04-12 15:55:39,297 DEBUG [StoreCloser-TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/4a04250de0db44d7bf907caeebc7d05c to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/4a04250de0db44d7bf907caeebc7d05c 2023-04-12 15:55:39,298 DEBUG [StoreCloser-TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/2406a5d42e7744ef8a79db3374b20945 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/2406a5d42e7744ef8a79db3374b20945 2023-04-12 15:55:39,299 DEBUG [StoreCloser-TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/bd87a964601b4b7caa7f139b5fb0af53 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/bd87a964601b4b7caa7f139b5fb0af53 2023-04-12 15:55:39,300 DEBUG [StoreCloser-TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/6a67a5c9dc714fb8bae990f7bc098db7 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/6a67a5c9dc714fb8bae990f7bc098db7 2023-04-12 15:55:39,302 DEBUG [StoreCloser-TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/2484a203716e4f4b96e33c5050ea7aa1 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/2484a203716e4f4b96e33c5050ea7aa1 2023-04-12 15:55:39,303 DEBUG [StoreCloser-TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/bb1d6b402f6646aaaad1bde2a3eb29ea to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/bb1d6b402f6646aaaad1bde2a3eb29ea 2023-04-12 15:55:39,304 DEBUG [StoreCloser-TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/da4fce448dce450987a91a7ed8dd7054 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/da4fce448dce450987a91a7ed8dd7054 2023-04-12 15:55:39,305 DEBUG [StoreCloser-TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/fe3f3d0e2bc24fa283df4bf8230c13af to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/fe3f3d0e2bc24fa283df4bf8230c13af 2023-04-12 15:55:39,306 DEBUG [StoreCloser-TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/50714ded97514e99be8da3c958b8a6fc to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/50714ded97514e99be8da3c958b8a6fc 2023-04-12 15:55:39,307 DEBUG [StoreCloser-TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/c9146fc2043c482c9522d42863f16176 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/c9146fc2043c482c9522d42863f16176 2023-04-12 15:55:39,308 DEBUG [StoreCloser-TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/769bc65e5d2e4daa9d03ea643e36dc6e to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/769bc65e5d2e4daa9d03ea643e36dc6e 2023-04-12 15:55:39,309 DEBUG [StoreCloser-TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/ebee2b3b4e7243a6aea7c34b0acd9f3d to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/ebee2b3b4e7243a6aea7c34b0acd9f3d 2023-04-12 15:55:39,311 DEBUG [StoreCloser-TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/b959bed4014944109878cdc0d49d2513 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/b959bed4014944109878cdc0d49d2513 2023-04-12 15:55:39,312 DEBUG [StoreCloser-TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/60eaba89351f477f81c0852406aaee26 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/60eaba89351f477f81c0852406aaee26 2023-04-12 15:55:39,314 DEBUG [StoreCloser-TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/e7dab56fe9474868a75f977b842924f8, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/013430c6da3a44b787d28110def3551f, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/d9f56c4fd8294b0b81536f3321e977b4, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/dfeaba1ef6924392b2ccbe0ed259afe8, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/d31c50b168fc4069b43b9418ccdcf464, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/a45629ce035b4ed492ff7439438f0696, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/6e80215966f14aa2ad8dba34bd33a971, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/b6e3ae710afb4438b471f301094b3067, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/24de0717013a46ba9a9d58a002b72cc9, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/2312d85197b64cae80b124896647d8f0, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/32ad9c3712c24119ac304e665319a379, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/76b6b15d792c4cd9aca15b9dd77c1336, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/eed82f35804f44ce8fe1a24eba3a28a9, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/439698319a144188a6561d466b958ff5, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/c402bb47e31643fbb80a0e8dd81c7a13, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/89a61f9dcfec467a921eb25e7ee1f656, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/938ee0d200b34f27becf6607a485f098, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/3e19d4e27579461eada55849c5cb3a3b, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/116bd8356ff34924999e86daf7547bbe, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/7a138e876f0247b4a2623c6fd91c8cd8, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/f93dc0e3f4a741baa0210b1351b16d45, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/3d00aa0b1734468f87b052ec78b67077, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/6a02019efb9940f2b47d98ec4583a962, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/755dc88ee6f142689159e9bebf7bc43a] to archive 2023-04-12 15:55:39,315 DEBUG [StoreCloser-TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-12 15:55:39,317 DEBUG [StoreCloser-TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/e7dab56fe9474868a75f977b842924f8 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/e7dab56fe9474868a75f977b842924f8 2023-04-12 15:55:39,318 DEBUG [StoreCloser-TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/013430c6da3a44b787d28110def3551f to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/013430c6da3a44b787d28110def3551f 2023-04-12 15:55:39,319 DEBUG [StoreCloser-TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/d9f56c4fd8294b0b81536f3321e977b4 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/d9f56c4fd8294b0b81536f3321e977b4 2023-04-12 15:55:39,321 DEBUG [StoreCloser-TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/dfeaba1ef6924392b2ccbe0ed259afe8 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/dfeaba1ef6924392b2ccbe0ed259afe8 2023-04-12 15:55:39,322 DEBUG [StoreCloser-TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/d31c50b168fc4069b43b9418ccdcf464 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/d31c50b168fc4069b43b9418ccdcf464 2023-04-12 15:55:39,324 DEBUG [StoreCloser-TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/a45629ce035b4ed492ff7439438f0696 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/a45629ce035b4ed492ff7439438f0696 2023-04-12 15:55:39,325 DEBUG [StoreCloser-TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/6e80215966f14aa2ad8dba34bd33a971 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/6e80215966f14aa2ad8dba34bd33a971 2023-04-12 15:55:39,326 DEBUG [StoreCloser-TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/b6e3ae710afb4438b471f301094b3067 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/b6e3ae710afb4438b471f301094b3067 2023-04-12 15:55:39,327 DEBUG [StoreCloser-TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/24de0717013a46ba9a9d58a002b72cc9 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/24de0717013a46ba9a9d58a002b72cc9 2023-04-12 15:55:39,329 DEBUG [StoreCloser-TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/2312d85197b64cae80b124896647d8f0 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/2312d85197b64cae80b124896647d8f0 2023-04-12 15:55:39,330 DEBUG [StoreCloser-TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/32ad9c3712c24119ac304e665319a379 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/32ad9c3712c24119ac304e665319a379 2023-04-12 15:55:39,331 DEBUG [StoreCloser-TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/76b6b15d792c4cd9aca15b9dd77c1336 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/76b6b15d792c4cd9aca15b9dd77c1336 2023-04-12 15:55:39,333 DEBUG [StoreCloser-TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/eed82f35804f44ce8fe1a24eba3a28a9 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/eed82f35804f44ce8fe1a24eba3a28a9 2023-04-12 15:55:39,334 DEBUG [StoreCloser-TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/439698319a144188a6561d466b958ff5 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/439698319a144188a6561d466b958ff5 2023-04-12 15:55:39,335 DEBUG [StoreCloser-TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/c402bb47e31643fbb80a0e8dd81c7a13 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/c402bb47e31643fbb80a0e8dd81c7a13 2023-04-12 15:55:39,336 DEBUG [StoreCloser-TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/89a61f9dcfec467a921eb25e7ee1f656 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/89a61f9dcfec467a921eb25e7ee1f656 2023-04-12 15:55:39,338 DEBUG [StoreCloser-TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/938ee0d200b34f27becf6607a485f098 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/938ee0d200b34f27becf6607a485f098 2023-04-12 15:55:39,339 DEBUG [StoreCloser-TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/3e19d4e27579461eada55849c5cb3a3b to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/3e19d4e27579461eada55849c5cb3a3b 2023-04-12 15:55:39,340 DEBUG [StoreCloser-TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/116bd8356ff34924999e86daf7547bbe to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/116bd8356ff34924999e86daf7547bbe 2023-04-12 15:55:39,341 DEBUG [StoreCloser-TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/7a138e876f0247b4a2623c6fd91c8cd8 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/7a138e876f0247b4a2623c6fd91c8cd8 2023-04-12 15:55:39,342 DEBUG [StoreCloser-TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/f93dc0e3f4a741baa0210b1351b16d45 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/f93dc0e3f4a741baa0210b1351b16d45 2023-04-12 15:55:39,348 DEBUG [StoreCloser-TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/3d00aa0b1734468f87b052ec78b67077 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/3d00aa0b1734468f87b052ec78b67077 2023-04-12 15:55:39,349 DEBUG [StoreCloser-TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/6a02019efb9940f2b47d98ec4583a962 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/6a02019efb9940f2b47d98ec4583a962 2023-04-12 15:55:39,350 DEBUG [StoreCloser-TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/755dc88ee6f142689159e9bebf7bc43a to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/755dc88ee6f142689159e9bebf7bc43a 2023-04-12 15:55:39,353 DEBUG [StoreCloser-TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/ba57ef6c23bd4c409856f688ab60c0f2, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/093dce0e9ed9469a83276e7d9a30ecc5, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/0de2554cf73244a8b34114410e9bd747, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/29139bb214e14f14af370063cbe612ef, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/7c26a31a9e8540409e1fa547f90c6ce0, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/fba63c08abfa4e8ba93b0e185a4c7a53, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/097b8c5ea1b647adafa8dd7c6f2768ba, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/622e866e408a4bea9ce2d7c712cff787, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/8a74a9c998c143e0a74e8b6a6cc3122e, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/b49c1225504e4e6498581338ff9dc928, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/bd1fe030bee8418d953f2c3742630f7f, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/63dd40f4fd6b4049b74c5e05ff33a4d0, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/193370cac8164b4498412057e1632419, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/aca9c9254bb84b1a84050a262795610a, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/189d5f660cd845c0a72abe3aa3ce7200, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/5decac26913342389fd06c008c80097a, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/93ed31725fca4baab843b39315bc5927, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/b568c95af0734d8cb036046fc3346665, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/bf8e37a5850d4275a01ca80f8b02939a, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/a0ee708199a44bfc9e33c2bab40ef921, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/9348125373ea4f269cb2df11cb5245fd, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/ac4fd241b7e24b2c8edf95cc91bd46cd, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/cf0b6bd86d5b4920b5aa57b14d245a80, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/5979e53a22e544dc98359ee996f7b887] to archive 2023-04-12 15:55:39,356 DEBUG [StoreCloser-TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-12 15:55:39,357 DEBUG [StoreCloser-TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/ba57ef6c23bd4c409856f688ab60c0f2 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/ba57ef6c23bd4c409856f688ab60c0f2 2023-04-12 15:55:39,358 DEBUG [StoreCloser-TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/093dce0e9ed9469a83276e7d9a30ecc5 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/093dce0e9ed9469a83276e7d9a30ecc5 2023-04-12 15:55:39,359 DEBUG [StoreCloser-TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/0de2554cf73244a8b34114410e9bd747 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/0de2554cf73244a8b34114410e9bd747 2023-04-12 15:55:39,360 DEBUG [StoreCloser-TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/29139bb214e14f14af370063cbe612ef to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/29139bb214e14f14af370063cbe612ef 2023-04-12 15:55:39,361 DEBUG [StoreCloser-TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/7c26a31a9e8540409e1fa547f90c6ce0 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/7c26a31a9e8540409e1fa547f90c6ce0 2023-04-12 15:55:39,362 DEBUG [StoreCloser-TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/fba63c08abfa4e8ba93b0e185a4c7a53 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/fba63c08abfa4e8ba93b0e185a4c7a53 2023-04-12 15:55:39,362 DEBUG [StoreCloser-TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/097b8c5ea1b647adafa8dd7c6f2768ba to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/097b8c5ea1b647adafa8dd7c6f2768ba 2023-04-12 15:55:39,363 DEBUG [StoreCloser-TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/622e866e408a4bea9ce2d7c712cff787 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/622e866e408a4bea9ce2d7c712cff787 2023-04-12 15:55:39,364 DEBUG [StoreCloser-TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/8a74a9c998c143e0a74e8b6a6cc3122e to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/8a74a9c998c143e0a74e8b6a6cc3122e 2023-04-12 15:55:39,365 DEBUG [StoreCloser-TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/b49c1225504e4e6498581338ff9dc928 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/b49c1225504e4e6498581338ff9dc928 2023-04-12 15:55:39,366 DEBUG [StoreCloser-TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/bd1fe030bee8418d953f2c3742630f7f to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/bd1fe030bee8418d953f2c3742630f7f 2023-04-12 15:55:39,367 DEBUG [StoreCloser-TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/63dd40f4fd6b4049b74c5e05ff33a4d0 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/63dd40f4fd6b4049b74c5e05ff33a4d0 2023-04-12 15:55:39,367 DEBUG [StoreCloser-TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/193370cac8164b4498412057e1632419 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/193370cac8164b4498412057e1632419 2023-04-12 15:55:39,368 DEBUG [StoreCloser-TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/aca9c9254bb84b1a84050a262795610a to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/aca9c9254bb84b1a84050a262795610a 2023-04-12 15:55:39,369 DEBUG [StoreCloser-TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/189d5f660cd845c0a72abe3aa3ce7200 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/189d5f660cd845c0a72abe3aa3ce7200 2023-04-12 15:55:39,370 DEBUG [StoreCloser-TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/5decac26913342389fd06c008c80097a to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/5decac26913342389fd06c008c80097a 2023-04-12 15:55:39,371 DEBUG [StoreCloser-TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/93ed31725fca4baab843b39315bc5927 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/93ed31725fca4baab843b39315bc5927 2023-04-12 15:55:39,371 DEBUG [StoreCloser-TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/b568c95af0734d8cb036046fc3346665 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/b568c95af0734d8cb036046fc3346665 2023-04-12 15:55:39,372 DEBUG [StoreCloser-TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/bf8e37a5850d4275a01ca80f8b02939a to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/bf8e37a5850d4275a01ca80f8b02939a 2023-04-12 15:55:39,373 DEBUG [StoreCloser-TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/a0ee708199a44bfc9e33c2bab40ef921 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/a0ee708199a44bfc9e33c2bab40ef921 2023-04-12 15:55:39,375 DEBUG [StoreCloser-TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/9348125373ea4f269cb2df11cb5245fd to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/9348125373ea4f269cb2df11cb5245fd 2023-04-12 15:55:39,376 DEBUG [StoreCloser-TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/ac4fd241b7e24b2c8edf95cc91bd46cd to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/ac4fd241b7e24b2c8edf95cc91bd46cd 2023-04-12 15:55:39,377 DEBUG [StoreCloser-TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/cf0b6bd86d5b4920b5aa57b14d245a80 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/cf0b6bd86d5b4920b5aa57b14d245a80 2023-04-12 15:55:39,378 DEBUG [StoreCloser-TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/5979e53a22e544dc98359ee996f7b887 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/5979e53a22e544dc98359ee996f7b887 2023-04-12 15:55:39,385 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/recovered.edits/367.seqid, newMaxSeqId=367, maxSeqId=4 2023-04-12 15:55:39,386 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a. 2023-04-12 15:55:39,387 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for d7d8c2ad266b62b02ebd2f4b9a678b1a: 2023-04-12 15:55:39,388 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed d7d8c2ad266b62b02ebd2f4b9a678b1a 2023-04-12 15:55:39,388 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=d7d8c2ad266b62b02ebd2f4b9a678b1a, regionState=CLOSED 2023-04-12 15:55:39,388 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681314939388"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681314939388"}]},"ts":"1681314939388"} 2023-04-12 15:55:39,390 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=45, resume processing ppid=44 2023-04-12 15:55:39,391 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=45, ppid=44, state=SUCCESS; CloseRegionProcedure d7d8c2ad266b62b02ebd2f4b9a678b1a, server=jenkins-hbase4.apache.org,38905,1681314818739 in 1.5390 sec 2023-04-12 15:55:39,392 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=44, resume processing ppid=43 2023-04-12 15:55:39,392 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=44, ppid=43, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=d7d8c2ad266b62b02ebd2f4b9a678b1a, UNASSIGN in 1.5430 sec 2023-04-12 15:55:39,392 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681314939392"}]},"ts":"1681314939392"} 2023-04-12 15:55:39,393 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-12 15:55:39,395 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-12 15:55:39,396 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=43, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.5560 sec 2023-04-12 15:55:39,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-12 15:55:39,948 INFO [Listener at localhost/43583] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 43 completed 2023-04-12 15:55:39,949 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-12 15:55:39,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure2.ProcedureExecutor(1029): Stored pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-12 15:55:39,952 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-12 15:55:39,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-04-12 15:55:39,953 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=46, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-12 15:55:39,957 DEBUG [HFileArchiver-9] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a 2023-04-12 15:55:39,959 DEBUG [HFileArchiver-9] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A, FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B, FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C, FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/recovered.edits] 2023-04-12 15:55:39,964 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/86c48de155cc4823805c76ad8a416243 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/86c48de155cc4823805c76ad8a416243 2023-04-12 15:55:39,966 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/c6ad090eeea84bc3bc21ee14e48cd9f1 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/c6ad090eeea84bc3bc21ee14e48cd9f1 2023-04-12 15:55:39,968 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/f6a5a5937d1b4231ace07f50cfb30422 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/A/f6a5a5937d1b4231ace07f50cfb30422 2023-04-12 15:55:39,971 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/1816f0145e2447c2bc08bf2815e16414 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/1816f0145e2447c2bc08bf2815e16414 2023-04-12 15:55:39,973 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/9ef6920c23fb430cae09b4853b86ac42 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/9ef6920c23fb430cae09b4853b86ac42 2023-04-12 15:55:39,974 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/ae0114a0023b4748b35404bd84f23620 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/B/ae0114a0023b4748b35404bd84f23620 2023-04-12 15:55:39,976 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/d86c6c8ae0254fecb11ecdfb21a06410 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/d86c6c8ae0254fecb11ecdfb21a06410 2023-04-12 15:55:39,977 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/dc8f3403d05a4df1be2974bec6187c2f to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/dc8f3403d05a4df1be2974bec6187c2f 2023-04-12 15:55:39,978 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/e1c5d6f99fc94fbdb0d8c1512e64ef22 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/C/e1c5d6f99fc94fbdb0d8c1512e64ef22 2023-04-12 15:55:39,982 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/recovered.edits/367.seqid to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a/recovered.edits/367.seqid 2023-04-12 15:55:39,982 DEBUG [HFileArchiver-9] backup.HFileArchiver(596): Deleted hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/d7d8c2ad266b62b02ebd2f4b9a678b1a 2023-04-12 15:55:39,982 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-12 15:55:39,983 DEBUG [PEWorker-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-12 15:55:39,983 DEBUG [PEWorker-4] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-12 15:55:39,985 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412037f90c7c1e14432ae213b8660263de4 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412037f90c7c1e14432ae213b8660263de4 2023-04-12 15:55:39,986 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041208d61b523f994a719ecc5df20453abdb to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041208d61b523f994a719ecc5df20453abdb 2023-04-12 15:55:39,987 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412191db7a04cf04a36b29612928a589f6b to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412191db7a04cf04a36b29612928a589f6b 2023-04-12 15:55:39,988 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304123ac86ad5277443838db3c23ed50538fe to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304123ac86ad5277443838db3c23ed50538fe 2023-04-12 15:55:39,989 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304124a686e237604451c812fe99aeb0e11d1 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304124a686e237604451c812fe99aeb0e11d1 2023-04-12 15:55:39,990 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304126694cb1707be417bb75f5a9005c175d1 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304126694cb1707be417bb75f5a9005c175d1 2023-04-12 15:55:39,991 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041268155fef94834d0ea6ad4e6480b39806 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041268155fef94834d0ea6ad4e6480b39806 2023-04-12 15:55:39,993 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304127690e4d3f9b742889f52a8880c9d541d to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304127690e4d3f9b742889f52a8880c9d541d 2023-04-12 15:55:39,994 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304127e5477d89335412ab1819492082c4795 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304127e5477d89335412ab1819492082c4795 2023-04-12 15:55:39,995 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304127fff971a256446758a82385ae5948185 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304127fff971a256446758a82385ae5948185 2023-04-12 15:55:39,996 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412888ab3999aac48ae91965a5b98583879 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412888ab3999aac48ae91965a5b98583879 2023-04-12 15:55:39,997 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412979dd48861c84db0b18208c62dc7b2cc to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412979dd48861c84db0b18208c62dc7b2cc 2023-04-12 15:55:39,998 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304129a34e3436bf247d4a5840ee58d54f519 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304129a34e3436bf247d4a5840ee58d54f519 2023-04-12 15:55:39,999 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304129a9e3e05f48f4d838f48c47e1c6ccb45 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304129a9e3e05f48f4d838f48c47e1c6ccb45 2023-04-12 15:55:40,000 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412a08092d048a6471daa35686cf6b36315 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412a08092d048a6471daa35686cf6b36315 2023-04-12 15:55:40,001 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412a57322dec6834032a01366ce18eca06f to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412a57322dec6834032a01366ce18eca06f 2023-04-12 15:55:40,002 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412b2a105df94c240f4b1a88930d71df0d4 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412b2a105df94c240f4b1a88930d71df0d4 2023-04-12 15:55:40,003 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412e2747e803b0c437692b659c1806a32cc to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412e2747e803b0c437692b659c1806a32cc 2023-04-12 15:55:40,004 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412e59baea78e5843e3bca6f7565c90f72f to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412e59baea78e5843e3bca6f7565c90f72f 2023-04-12 15:55:40,005 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412e9d885533bdc440fa234e7f0948fcd7e to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412e9d885533bdc440fa234e7f0948fcd7e 2023-04-12 15:55:40,006 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412fee0eb20a6e44fae9a5d9ddd6627b0e8 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412fee0eb20a6e44fae9a5d9ddd6627b0e8 2023-04-12 15:55:40,007 DEBUG [PEWorker-4] backup.HFileArchiver(596): Deleted hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-12 15:55:40,009 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=46, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-12 15:55:40,010 WARN [PEWorker-4] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-12 15:55:40,012 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-12 15:55:40,013 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=46, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-12 15:55:40,013 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-12 15:55:40,013 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681314940013"}]},"ts":"9223372036854775807"} 2023-04-12 15:55:40,014 INFO [PEWorker-4] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-12 15:55:40,014 DEBUG [PEWorker-4] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => d7d8c2ad266b62b02ebd2f4b9a678b1a, NAME => 'TestAcidGuarantees,,1681314912262.d7d8c2ad266b62b02ebd2f4b9a678b1a.', STARTKEY => '', ENDKEY => ''}] 2023-04-12 15:55:40,014 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-12 15:55:40,015 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681314940015"}]},"ts":"9223372036854775807"} 2023-04-12 15:55:40,016 INFO [PEWorker-4] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-12 15:55:40,019 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-04-12 15:55:40,023 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(130): Finished pid=46, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-12 15:55:40,024 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=46, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 74 msec 2023-04-12 15:55:40,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-04-12 15:55:40,054 INFO [Listener at localhost/43583] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 46 completed 2023-04-12 15:55:40,068 INFO [Listener at localhost/43583] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=229 (was 224) - Thread LEAK? -, OpenFileDescriptor=545 (was 534) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=345 (was 356), ProcessCount=169 (was 169), AvailableMemoryMB=1259 (was 523) - AvailableMemoryMB LEAK? - 2023-04-12 15:55:40,083 INFO [Listener at localhost/43583] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=229, OpenFileDescriptor=545, MaxFileDescriptor=60000, SystemLoadAverage=345, ProcessCount=169, AvailableMemoryMB=1259 2023-04-12 15:55:40,085 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-12 15:55:40,085 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-12 15:55:40,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure2.ProcedureExecutor(1029): Stored pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-12 15:55:40,087 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-12 15:55:40,088 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 47 2023-04-12 15:55:40,088 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-12 15:55:40,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-12 15:55:40,089 DEBUG [HFileArchiver-10] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530 2023-04-12 15:55:40,090 DEBUG [HFileArchiver-10] backup.HFileArchiver(153): Directory hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530 empty. 2023-04-12 15:55:40,090 DEBUG [HFileArchiver-10] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530 2023-04-12 15:55:40,090 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-12 15:55:40,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-12 15:55:40,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-12 15:55:40,498 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-12 15:55:40,500 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 4df381dfb8e2792d3aca65faf2673530, NAME => 'TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp 2023-04-12 15:55:40,518 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-12 15:55:40,518 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 4df381dfb8e2792d3aca65faf2673530, disabling compactions & flushes 2023-04-12 15:55:40,518 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:40,518 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:40,518 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. after waiting 0 ms 2023-04-12 15:55:40,518 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:40,518 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:40,518 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 4df381dfb8e2792d3aca65faf2673530: 2023-04-12 15:55:40,520 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-12 15:55:40,521 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681314940521"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681314940521"}]},"ts":"1681314940521"} 2023-04-12 15:55:40,522 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-12 15:55:40,523 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-12 15:55:40,523 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681314940523"}]},"ts":"1681314940523"} 2023-04-12 15:55:40,523 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-12 15:55:40,527 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=4df381dfb8e2792d3aca65faf2673530, ASSIGN}] 2023-04-12 15:55:40,529 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=4df381dfb8e2792d3aca65faf2673530, ASSIGN 2023-04-12 15:55:40,529 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=4df381dfb8e2792d3aca65faf2673530, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,38905,1681314818739; forceNewPlan=false, retain=false 2023-04-12 15:55:40,681 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=4df381dfb8e2792d3aca65faf2673530, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:40,681 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681314940680"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681314940680"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681314940680"}]},"ts":"1681314940680"} 2023-04-12 15:55:40,682 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=49, ppid=48, state=RUNNABLE; OpenRegionProcedure 4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739}] 2023-04-12 15:55:40,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-12 15:55:40,837 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:40,837 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 4df381dfb8e2792d3aca65faf2673530, NAME => 'TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.', STARTKEY => '', ENDKEY => ''} 2023-04-12 15:55:40,838 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 4df381dfb8e2792d3aca65faf2673530 2023-04-12 15:55:40,838 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-12 15:55:40,838 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 4df381dfb8e2792d3aca65faf2673530 2023-04-12 15:55:40,838 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 4df381dfb8e2792d3aca65faf2673530 2023-04-12 15:55:40,839 INFO [StoreOpener-4df381dfb8e2792d3aca65faf2673530-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 4df381dfb8e2792d3aca65faf2673530 2023-04-12 15:55:40,840 DEBUG [StoreOpener-4df381dfb8e2792d3aca65faf2673530-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A 2023-04-12 15:55:40,840 DEBUG [StoreOpener-4df381dfb8e2792d3aca65faf2673530-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A 2023-04-12 15:55:40,841 INFO [StoreOpener-4df381dfb8e2792d3aca65faf2673530-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-12 15:55:40,841 INFO [StoreOpener-4df381dfb8e2792d3aca65faf2673530-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4df381dfb8e2792d3aca65faf2673530 columnFamilyName A 2023-04-12 15:55:40,842 INFO [StoreOpener-4df381dfb8e2792d3aca65faf2673530-1] regionserver.HStore(310): Store=4df381dfb8e2792d3aca65faf2673530/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 15:55:40,842 INFO [StoreOpener-4df381dfb8e2792d3aca65faf2673530-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 4df381dfb8e2792d3aca65faf2673530 2023-04-12 15:55:40,843 DEBUG [StoreOpener-4df381dfb8e2792d3aca65faf2673530-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B 2023-04-12 15:55:40,843 DEBUG [StoreOpener-4df381dfb8e2792d3aca65faf2673530-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B 2023-04-12 15:55:40,844 INFO [StoreOpener-4df381dfb8e2792d3aca65faf2673530-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-12 15:55:40,844 INFO [StoreOpener-4df381dfb8e2792d3aca65faf2673530-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4df381dfb8e2792d3aca65faf2673530 columnFamilyName B 2023-04-12 15:55:40,844 INFO [StoreOpener-4df381dfb8e2792d3aca65faf2673530-1] regionserver.HStore(310): Store=4df381dfb8e2792d3aca65faf2673530/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 15:55:40,844 INFO [StoreOpener-4df381dfb8e2792d3aca65faf2673530-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 4df381dfb8e2792d3aca65faf2673530 2023-04-12 15:55:40,846 DEBUG [StoreOpener-4df381dfb8e2792d3aca65faf2673530-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C 2023-04-12 15:55:40,846 DEBUG [StoreOpener-4df381dfb8e2792d3aca65faf2673530-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C 2023-04-12 15:55:40,846 INFO [StoreOpener-4df381dfb8e2792d3aca65faf2673530-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-12 15:55:40,846 INFO [StoreOpener-4df381dfb8e2792d3aca65faf2673530-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4df381dfb8e2792d3aca65faf2673530 columnFamilyName C 2023-04-12 15:55:40,846 INFO [StoreOpener-4df381dfb8e2792d3aca65faf2673530-1] regionserver.HStore(310): Store=4df381dfb8e2792d3aca65faf2673530/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 15:55:40,847 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:40,847 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530 2023-04-12 15:55:40,848 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530 2023-04-12 15:55:40,849 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-12 15:55:40,850 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 4df381dfb8e2792d3aca65faf2673530 2023-04-12 15:55:40,852 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-12 15:55:40,852 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 4df381dfb8e2792d3aca65faf2673530; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69887403, jitterRate=0.04140345752239227}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-12 15:55:40,852 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 4df381dfb8e2792d3aca65faf2673530: 2023-04-12 15:55:40,853 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530., pid=49, masterSystemTime=1681314940834 2023-04-12 15:55:40,854 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:40,854 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:40,855 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=4df381dfb8e2792d3aca65faf2673530, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:40,855 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681314940854"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681314940854"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681314940854"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681314940854"}]},"ts":"1681314940854"} 2023-04-12 15:55:40,857 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=49, resume processing ppid=48 2023-04-12 15:55:40,857 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=49, ppid=48, state=SUCCESS; OpenRegionProcedure 4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 in 174 msec 2023-04-12 15:55:40,858 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=48, resume processing ppid=47 2023-04-12 15:55:40,858 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=48, ppid=47, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=4df381dfb8e2792d3aca65faf2673530, ASSIGN in 330 msec 2023-04-12 15:55:40,859 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-12 15:55:40,859 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681314940859"}]},"ts":"1681314940859"} 2023-04-12 15:55:40,860 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-12 15:55:40,863 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-12 15:55:40,864 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=47, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 778 msec 2023-04-12 15:55:41,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-12 15:55:41,192 INFO [Listener at localhost/43583] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 47 completed 2023-04-12 15:55:41,193 DEBUG [Listener at localhost/43583] zookeeper.ReadOnlyZKClient(139): Connect 0x15880ffb to 127.0.0.1:60154 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 15:55:41,199 DEBUG [Listener at localhost/43583] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@20ed9117, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 15:55:41,200 DEBUG [Listener at localhost/43583] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 15:55:41,202 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55246, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 15:55:41,203 DEBUG [Listener at localhost/43583] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-12 15:55:41,204 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37262, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-12 15:55:41,206 DEBUG [Listener at localhost/43583] zookeeper.ReadOnlyZKClient(139): Connect 0x052203cb to 127.0.0.1:60154 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 15:55:41,210 DEBUG [Listener at localhost/43583] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@50002343, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 15:55:41,210 DEBUG [Listener at localhost/43583] zookeeper.ReadOnlyZKClient(139): Connect 0x420f945a to 127.0.0.1:60154 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 15:55:41,218 DEBUG [Listener at localhost/43583] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5726fb8b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 15:55:41,218 DEBUG [Listener at localhost/43583] zookeeper.ReadOnlyZKClient(139): Connect 0x51b8df2a to 127.0.0.1:60154 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 15:55:41,226 DEBUG [Listener at localhost/43583] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6f2b3c03, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 15:55:41,226 DEBUG [Listener at localhost/43583] zookeeper.ReadOnlyZKClient(139): Connect 0x67346c47 to 127.0.0.1:60154 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 15:55:41,231 DEBUG [Listener at localhost/43583] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@70e2be04, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 15:55:41,231 DEBUG [Listener at localhost/43583] zookeeper.ReadOnlyZKClient(139): Connect 0x426cb82a to 127.0.0.1:60154 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 15:55:41,235 DEBUG [Listener at localhost/43583] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@67ea274, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 15:55:41,236 DEBUG [Listener at localhost/43583] zookeeper.ReadOnlyZKClient(139): Connect 0x78592df0 to 127.0.0.1:60154 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 15:55:41,245 DEBUG [Listener at localhost/43583] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@b475d8d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 15:55:41,245 DEBUG [Listener at localhost/43583] zookeeper.ReadOnlyZKClient(139): Connect 0x659c68ab to 127.0.0.1:60154 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 15:55:41,251 DEBUG [Listener at localhost/43583] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@549bef45, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 15:55:41,251 DEBUG [Listener at localhost/43583] zookeeper.ReadOnlyZKClient(139): Connect 0x680e773e to 127.0.0.1:60154 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 15:55:41,260 DEBUG [Listener at localhost/43583] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6ed183a4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 15:55:41,261 DEBUG [Listener at localhost/43583] zookeeper.ReadOnlyZKClient(139): Connect 0x0b8f652e to 127.0.0.1:60154 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 15:55:41,271 DEBUG [Listener at localhost/43583] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@10a344e6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 15:55:41,272 DEBUG [Listener at localhost/43583] zookeeper.ReadOnlyZKClient(139): Connect 0x4ac98310 to 127.0.0.1:60154 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 15:55:41,280 DEBUG [Listener at localhost/43583] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3a9f3972, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 15:55:41,288 DEBUG [hconnection-0x1614274a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 15:55:41,301 DEBUG [hconnection-0x5bb64324-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 15:55:41,303 DEBUG [hconnection-0x14496706-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 15:55:41,304 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55266, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 15:55:41,304 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:41,308 DEBUG [hconnection-0x421ad59c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 15:55:41,308 DEBUG [hconnection-0x7f93d9fd-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 15:55:41,309 DEBUG [hconnection-0x31a5fa3a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 15:55:41,309 DEBUG [hconnection-0x7df3d1b7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 15:55:41,309 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55260, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 15:55:41,313 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55280, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 15:55:41,314 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55306, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 15:55:41,316 DEBUG [hconnection-0x5cf3fb94-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 15:55:41,316 DEBUG [hconnection-0x3eefb34a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 15:55:41,316 DEBUG [hconnection-0x6289bf6a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 15:55:41,316 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55322, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 15:55:41,317 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55290, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 15:55:41,317 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55336, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 15:55:41,317 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55340, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 15:55:41,320 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55354, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 15:55:41,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on 4df381dfb8e2792d3aca65faf2673530 2023-04-12 15:55:41,329 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4df381dfb8e2792d3aca65faf2673530 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-12 15:55:41,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:41,329 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=A 2023-04-12 15:55:41,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:41,329 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:41,329 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:41,329 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:41,329 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=B 2023-04-12 15:55:41,329 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:41,329 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:41,329 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:41,329 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=C 2023-04-12 15:55:41,329 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:41,330 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:41,330 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:41,337 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:41,337 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:41,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:41,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:41,338 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:41,338 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:41,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:41,338 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:41,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:41,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:41,339 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:41,345 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:41,345 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:41,345 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:41,345 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:41,345 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:41,345 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55370, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 15:55:41,345 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:41,345 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:41,346 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:41,346 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. started... 2023-04-12 15:55:41,346 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. as already flushing 2023-04-12 15:55:41,346 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:41,346 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:41,346 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:41,346 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:41,348 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:41,348 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:41,362 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:41,362 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:41,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:41,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:41,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:41,362 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:41,362 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:41,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:41,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:41,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:41,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:41,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:41,364 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:41,364 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@fc1734[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:41,364 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:41,364 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:41,366 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:41,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:41,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:41,366 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:41,366 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:41,366 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:41,366 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:41,366 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:41,369 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:41,369 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:41,369 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:41,369 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:41,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:41,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:41,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:41,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:41,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:41,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:41,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:41,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:41,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:41,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:41,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:41,373 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:41,373 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:41,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55306 deadline: 1681315001365, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:41,373 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:41,373 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:41,373 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:41,373 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:41,373 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:41,373 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:41,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 5 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55340 deadline: 1681315001365, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:41,374 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:41,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55336 deadline: 1681315001365, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:41,375 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:41,375 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:41,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:41,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:41,375 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:41,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:41,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:41,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:41,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:41,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:41,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:41,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:41,377 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:41,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:41,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:41,379 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:41,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 4 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55322 deadline: 1681315001372, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:41,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:41,379 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:41,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:41,379 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:41,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:41,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55260 deadline: 1681315001374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:41,380 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/A/d3919cdd5ebc40be8714793670797f48 2023-04-12 15:55:41,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:41,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:41,383 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:41,383 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:41,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:41,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:41,383 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:41,383 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:41,383 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:41,383 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:41,383 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:41,383 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:41,383 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:41,383 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:41,383 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:41,383 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:41,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:41,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:41,384 DEBUG [Thread-3125] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:41,384 DEBUG [Thread-3125] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:41,384 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:41,475 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:41,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55306 deadline: 1681315001474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:41,475 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:41,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55340 deadline: 1681315001475, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:41,476 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:41,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55336 deadline: 1681315001475, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:41,484 DEBUG [Thread-3125] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:41,484 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:41,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55322 deadline: 1681315001480, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:41,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:41,485 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:41,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55260 deadline: 1681315001485, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:41,486 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:41,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:41,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:41,491 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:41,491 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:41,491 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:41,491 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:41,492 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:41,492 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:41,493 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:41,493 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:41,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:41,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:41,494 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:41,494 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:41,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:41,494 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:41,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:41,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:41,495 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:41,501 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:41,501 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:41,501 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:41,501 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:41,501 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:41,502 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:41,502 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:41,504 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:41,504 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. started... 2023-04-12 15:55:41,511 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. as already flushing 2023-04-12 15:55:41,511 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:41,511 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:41,511 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:41,511 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:41,515 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:41,515 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:41,517 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:41,517 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:41,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:41,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:41,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:41,518 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:41,518 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:41,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:41,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:41,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:41,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:41,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:41,519 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:41,519 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ac10a9e[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:41,519 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:41,519 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:41,521 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:41,521 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:41,521 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:41,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:41,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:41,521 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:41,521 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:41,521 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:41,523 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:41,523 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:41,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:41,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:41,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:41,523 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:41,524 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:41,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:41,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:41,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:41,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:41,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:41,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:41,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:41,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:41,526 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:41,526 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:41,526 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:41,526 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:41,526 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:41,526 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:41,527 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:41,527 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:41,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:41,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:41,528 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:41,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:41,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:41,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:41,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:41,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:41,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:41,530 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:41,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:41,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:41,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:41,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:41,532 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:41,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:41,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:41,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:41,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:41,537 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:41,537 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:41,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:41,538 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:41,538 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:41,538 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:41,538 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:41,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:41,537 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:41,537 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:41,538 DEBUG [Thread-3125] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:41,538 DEBUG [Thread-3125] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:41,538 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:41,538 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:41,538 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:41,538 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:41,538 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:41,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:41,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:41,638 DEBUG [Thread-3125] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:41,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:41,640 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:41,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:41,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:41,650 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:41,650 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:41,650 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:41,650 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:41,650 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:41,650 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:41,651 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:41,651 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:41,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:41,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:41,652 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:41,652 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:41,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:41,652 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:41,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:41,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:41,653 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:41,663 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:41,663 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:41,663 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:41,663 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:41,663 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:41,668 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:41,668 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:41,668 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:41,668 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. started... 2023-04-12 15:55:41,668 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. as already flushing 2023-04-12 15:55:41,668 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:41,668 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:41,668 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:41,668 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:41,674 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:41,674 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:41,677 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:41,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55336 deadline: 1681315001677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:41,678 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:41,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55340 deadline: 1681315001678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:41,678 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:41,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55306 deadline: 1681315001678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:41,681 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:41,681 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:41,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:41,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:41,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:41,681 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:41,681 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:41,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:41,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:41,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:41,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:41,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:41,682 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:41,682 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2320c21a[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:41,682 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:41,682 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:41,683 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:41,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:41,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:41,684 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:41,684 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:41,684 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:41,684 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:41,684 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:41,685 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:41,685 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:41,685 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:41,685 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:41,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:41,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:41,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:41,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:41,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:41,686 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:41,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55260 deadline: 1681315001686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:41,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:41,687 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:41,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:41,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55322 deadline: 1681315001687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:41,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:41,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:41,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:41,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:41,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:41,688 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:41,688 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:41,688 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:41,688 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:41,688 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:41,690 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:41,690 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:41,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:41,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:41,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:41,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:41,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:41,690 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:41,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:41,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:41,691 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:41,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:41,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:41,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:41,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:41,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:41,692 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:41,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:41,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:41,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:41,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:41,695 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:41,695 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:41,695 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:41,695 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:41,695 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:41,695 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:41,695 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:41,695 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:41,695 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:41,695 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:41,695 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:41,696 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:41,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:41,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:41,696 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:41,696 DEBUG [Thread-3125] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:41,696 DEBUG [Thread-3125] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:41,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:41,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:41,796 DEBUG [Thread-3125] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:41,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:41,798 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:41,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:41,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:41,800 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:41,800 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:41,800 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:41,800 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:41,800 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:41,800 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:41,802 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:41,802 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:41,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:41,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:41,802 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:41,802 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:41,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:41,802 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:41,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:41,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:41,803 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:41,803 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:41,803 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:41,803 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:41,803 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:41,803 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:41,804 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:41,804 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:41,804 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:41,804 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. started... 2023-04-12 15:55:41,804 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. as already flushing 2023-04-12 15:55:41,804 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:41,804 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:41,804 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:41,804 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:41,804 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:41,804 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:41,806 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:41,806 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:41,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:41,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:41,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:41,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:41,806 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:41,807 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:41,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:41,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:41,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:41,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:41,808 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:41,808 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@aa15d27[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:41,808 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:41,808 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:41,809 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:41,809 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:41,816 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:41,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:41,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:41,816 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:41,816 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:41,816 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:41,818 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:41,818 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:41,818 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:41,818 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:41,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:41,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:41,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:41,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:41,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:41,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:41,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:41,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:41,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:41,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:41,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:41,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:41,821 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:41,821 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:41,821 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:41,821 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:41,821 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:41,822 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:41,823 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:41,823 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:41,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:41,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:41,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:41,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:41,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:41,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:41,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:41,823 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:41,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:41,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:41,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:41,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:41,829 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:41,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:41,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:41,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:41,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:41,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:41,832 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:41,833 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:41,833 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:41,833 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:41,833 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:41,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:41,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:41,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:41,833 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:41,833 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:41,833 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:41,833 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:41,833 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:41,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:41,833 DEBUG [Thread-3125] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:41,833 DEBUG [Thread-3125] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:41,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:41,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:41,834 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:41,866 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/B/79eb74a3bda743dfb58a6637261f6e1a 2023-04-12 15:55:41,904 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/C/7c0ede0be0e64184a182cb2c16de4796 2023-04-12 15:55:41,908 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/A/d3919cdd5ebc40be8714793670797f48 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/d3919cdd5ebc40be8714793670797f48 2023-04-12 15:55:41,913 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/d3919cdd5ebc40be8714793670797f48, entries=150, sequenceid=13, filesize=11.6 K 2023-04-12 15:55:41,916 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/B/79eb74a3bda743dfb58a6637261f6e1a as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/79eb74a3bda743dfb58a6637261f6e1a 2023-04-12 15:55:41,922 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/79eb74a3bda743dfb58a6637261f6e1a, entries=150, sequenceid=13, filesize=11.6 K 2023-04-12 15:55:41,923 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/C/7c0ede0be0e64184a182cb2c16de4796 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/7c0ede0be0e64184a182cb2c16de4796 2023-04-12 15:55:41,927 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/7c0ede0be0e64184a182cb2c16de4796, entries=150, sequenceid=13, filesize=11.6 K 2023-04-12 15:55:41,927 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 4df381dfb8e2792d3aca65faf2673530 in 599ms, sequenceid=13, compaction requested=false 2023-04-12 15:55:41,927 DEBUG [MemStoreFlusher.0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-12 15:55:41,929 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4df381dfb8e2792d3aca65faf2673530: 2023-04-12 15:55:41,933 DEBUG [Thread-3125] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:41,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:41,935 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:41,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:41,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:41,937 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:41,937 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:41,937 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:41,937 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:41,937 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:41,937 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:41,939 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:41,939 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:41,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:41,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:41,939 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:41,939 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:41,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:41,939 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:41,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:41,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:41,941 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:41,941 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:41,941 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:41,941 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:41,941 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:41,941 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:41,941 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:41,941 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:41,941 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:41,941 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. started... 2023-04-12 15:55:41,941 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4df381dfb8e2792d3aca65faf2673530 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-12 15:55:41,942 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=A 2023-04-12 15:55:41,942 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:41,942 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=B 2023-04-12 15:55:41,942 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:41,942 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=C 2023-04-12 15:55:41,942 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:41,983 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. as already flushing 2023-04-12 15:55:41,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on 4df381dfb8e2792d3aca65faf2673530 2023-04-12 15:55:42,000 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:42,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55340 deadline: 1681315001997, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:42,007 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:42,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55336 deadline: 1681315001997, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:42,009 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:42,009 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=33 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/A/ea1d95daec7642c7afdd4fb099f40516 2023-04-12 15:55:42,009 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:42,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55260 deadline: 1681315001999, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:42,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55322 deadline: 1681315001999, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:42,009 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:42,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55306 deadline: 1681315002001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:42,035 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=33 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/B/055b65b5857a4098a881ede417a4cedf 2023-04-12 15:55:42,106 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:42,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55340 deadline: 1681315002102, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:42,112 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:42,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55336 deadline: 1681315002108, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:42,114 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:42,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55260 deadline: 1681315002111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:42,114 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:42,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55306 deadline: 1681315002111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:42,115 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:42,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55322 deadline: 1681315002112, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:42,311 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:42,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55340 deadline: 1681315002308, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:42,318 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:42,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55336 deadline: 1681315002313, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:42,320 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:42,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55306 deadline: 1681315002315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:42,320 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:42,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55260 deadline: 1681315002316, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:42,320 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:42,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55322 deadline: 1681315002317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:42,465 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=33 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/C/cd6a46adcece4549aa04f48ee04f698f 2023-04-12 15:55:42,471 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/A/ea1d95daec7642c7afdd4fb099f40516 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/ea1d95daec7642c7afdd4fb099f40516 2023-04-12 15:55:42,475 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/ea1d95daec7642c7afdd4fb099f40516, entries=150, sequenceid=33, filesize=11.6 K 2023-04-12 15:55:42,476 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/B/055b65b5857a4098a881ede417a4cedf as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/055b65b5857a4098a881ede417a4cedf 2023-04-12 15:55:42,480 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/055b65b5857a4098a881ede417a4cedf, entries=150, sequenceid=33, filesize=11.6 K 2023-04-12 15:55:42,480 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/C/cd6a46adcece4549aa04f48ee04f698f as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/cd6a46adcece4549aa04f48ee04f698f 2023-04-12 15:55:42,490 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/cd6a46adcece4549aa04f48ee04f698f, entries=150, sequenceid=33, filesize=11.6 K 2023-04-12 15:55:42,491 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 4df381dfb8e2792d3aca65faf2673530 in 550ms, sequenceid=33, compaction requested=false 2023-04-12 15:55:42,492 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4df381dfb8e2792d3aca65faf2673530: 2023-04-12 15:55:42,492 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:42,492 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:42,493 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:42,493 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:42,493 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:42,493 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:42,495 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:42,495 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:42,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:42,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:42,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:42,495 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:42,495 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:42,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:42,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:42,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:42,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:42,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:42,497 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:42,497 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a84aef0[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:42,497 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:42,497 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:42,498 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:42,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:42,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:42,498 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:42,498 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:42,498 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:42,498 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:42,498 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:42,499 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:42,499 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:42,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:42,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:42,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:42,499 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:42,500 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:42,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:42,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:42,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:42,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:42,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:42,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:42,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:42,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:42,502 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:42,502 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:42,502 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:42,502 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:42,502 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:42,502 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:42,503 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:42,503 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:42,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:42,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:42,504 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:42,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:42,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:42,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:42,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:42,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:42,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:42,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:42,505 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:42,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:42,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:42,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:42,507 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:42,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:42,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:42,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:42,512 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:42,512 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:42,512 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:42,512 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:42,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:42,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:42,512 DEBUG [Thread-3125] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:42,512 DEBUG [Thread-3125] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:42,512 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:42,512 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:42,512 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:42,512 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:42,512 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:42,512 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:42,512 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:42,513 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:42,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:42,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:42,513 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:42,612 DEBUG [Thread-3125] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:42,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:42,615 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:42,615 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4df381dfb8e2792d3aca65faf2673530 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-12 15:55:42,615 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=A 2023-04-12 15:55:42,615 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:42,615 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=B 2023-04-12 15:55:42,615 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:42,615 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=C 2023-04-12 15:55:42,615 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:42,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on 4df381dfb8e2792d3aca65faf2673530 2023-04-12 15:55:42,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:42,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:42,621 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:42,621 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:42,621 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:42,621 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:42,622 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:42,623 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:42,625 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:42,625 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:42,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:42,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:42,626 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:42,626 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:42,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:42,627 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:42,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:42,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:42,627 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:42,628 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:42,628 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:42,628 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:42,628 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:42,628 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:42,633 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:42,633 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:42,633 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:42,633 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. started... 2023-04-12 15:55:42,633 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. as already flushing 2023-04-12 15:55:42,633 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:42,633 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:42,633 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:42,633 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:42,644 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:42,644 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:42,646 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:42,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55306 deadline: 1681315002639, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:42,647 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=46 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/A/725b98bfe966453585d85de72d01c8ca 2023-04-12 15:55:42,647 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:42,647 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:42,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:42,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:42,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:42,647 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:42,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55322 deadline: 1681315002640, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:42,648 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:42,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55336 deadline: 1681315002641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:42,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:42,649 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:42,649 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:42,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:42,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:42,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:42,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:42,651 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:42,651 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7883ccc4[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:42,651 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:42,651 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:42,652 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:42,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:42,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:42,652 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:42,652 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:42,653 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:42,653 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:42,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55260 deadline: 1681315002645, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:42,653 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:42,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55340 deadline: 1681315002646, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:42,654 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:42,654 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:42,659 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:42,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:42,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:42,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:42,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:42,659 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:42,659 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:42,659 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:42,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:42,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:42,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:42,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:42,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:42,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:42,662 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=46 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/B/fed29ce026994db99cea3de2425667d3 2023-04-12 15:55:42,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:42,662 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:42,662 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:42,662 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:42,662 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:42,662 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:42,662 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:42,664 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:42,664 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:42,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:42,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:42,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:42,664 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:42,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:42,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:42,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:42,665 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:42,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:42,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:42,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:42,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:42,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:42,666 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:42,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:42,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:42,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:42,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:42,669 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:42,669 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:42,669 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:42,669 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:42,669 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:42,669 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:42,669 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:42,669 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:42,669 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:42,669 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:42,669 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:42,669 DEBUG [Thread-3125] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:42,669 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:42,669 DEBUG [Thread-3125] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:42,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:42,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:42,670 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:42,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:42,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:42,676 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=46 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/C/4993656bbde34d1096a4c893bb940815 2023-04-12 15:55:42,681 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/A/725b98bfe966453585d85de72d01c8ca as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/725b98bfe966453585d85de72d01c8ca 2023-04-12 15:55:42,684 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/725b98bfe966453585d85de72d01c8ca, entries=200, sequenceid=46, filesize=13.9 K 2023-04-12 15:55:42,689 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/B/fed29ce026994db99cea3de2425667d3 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/fed29ce026994db99cea3de2425667d3 2023-04-12 15:55:42,694 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/fed29ce026994db99cea3de2425667d3, entries=150, sequenceid=46, filesize=11.6 K 2023-04-12 15:55:42,695 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/C/4993656bbde34d1096a4c893bb940815 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/4993656bbde34d1096a4c893bb940815 2023-04-12 15:55:42,706 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/4993656bbde34d1096a4c893bb940815, entries=150, sequenceid=46, filesize=11.6 K 2023-04-12 15:55:42,707 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 4df381dfb8e2792d3aca65faf2673530 in 92ms, sequenceid=46, compaction requested=true 2023-04-12 15:55:42,707 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4df381dfb8e2792d3aca65faf2673530: 2023-04-12 15:55:42,707 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:55:42,707 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 15:55:42,707 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:55:42,707 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 15:55:42,708 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:55:42,710 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37998 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 15:55:42,711 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1912): 4df381dfb8e2792d3aca65faf2673530/A is initiating minor compaction (all files) 2023-04-12 15:55:42,711 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 15:55:42,711 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4df381dfb8e2792d3aca65faf2673530/A in TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:42,711 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1912): 4df381dfb8e2792d3aca65faf2673530/B is initiating minor compaction (all files) 2023-04-12 15:55:42,711 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/d3919cdd5ebc40be8714793670797f48, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/ea1d95daec7642c7afdd4fb099f40516, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/725b98bfe966453585d85de72d01c8ca] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp, totalSize=37.1 K 2023-04-12 15:55:42,711 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4df381dfb8e2792d3aca65faf2673530/B in TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:42,711 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/79eb74a3bda743dfb58a6637261f6e1a, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/055b65b5857a4098a881ede417a4cedf, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/fed29ce026994db99cea3de2425667d3] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp, totalSize=34.8 K 2023-04-12 15:55:42,711 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting d3919cdd5ebc40be8714793670797f48, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1681314941321 2023-04-12 15:55:42,711 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 79eb74a3bda743dfb58a6637261f6e1a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1681314941321 2023-04-12 15:55:42,712 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting ea1d95daec7642c7afdd4fb099f40516, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1681314941362 2023-04-12 15:55:42,712 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 055b65b5857a4098a881ede417a4cedf, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1681314941362 2023-04-12 15:55:42,712 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 725b98bfe966453585d85de72d01c8ca, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1681314941996 2023-04-12 15:55:42,712 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting fed29ce026994db99cea3de2425667d3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1681314941996 2023-04-12 15:55:42,733 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4df381dfb8e2792d3aca65faf2673530#A#compaction#407 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:55:42,737 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] throttle.PressureAwareThroughputController(145): 4df381dfb8e2792d3aca65faf2673530#B#compaction#408 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:55:42,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on 4df381dfb8e2792d3aca65faf2673530 2023-04-12 15:55:42,758 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4df381dfb8e2792d3aca65faf2673530 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-12 15:55:42,759 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681314942628, max timestamp=1681314942756Num uniques 859; 2023-04-12 15:55:42,759 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-12 15:55:42,759 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681314942628, max timestamp=1681314942756Num uniques 859; 2023-04-12 15:55:42,759 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-12 15:55:42,759 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=A 2023-04-12 15:55:42,760 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:55:42,760 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=B 2023-04-12 15:55:42,760 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:55:42,760 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=C 2023-04-12 15:55:42,760 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:55:42,769 DEBUG [Thread-3125] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:42,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:42,772 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:42,780 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/B/7a421934a8794a71a2e53fc012068644 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/7a421934a8794a71a2e53fc012068644 2023-04-12 15:55:42,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:42,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:42,784 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:42,784 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:42,784 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:42,784 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:42,785 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:42,785 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:42,786 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4df381dfb8e2792d3aca65faf2673530/B of 4df381dfb8e2792d3aca65faf2673530 into 7a421934a8794a71a2e53fc012068644(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:55:42,786 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4df381dfb8e2792d3aca65faf2673530: 2023-04-12 15:55:42,786 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530., storeName=4df381dfb8e2792d3aca65faf2673530/B, priority=13, startTime=1681314942707; duration=0sec 2023-04-12 15:55:42,786 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:55:42,786 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 15:55:42,786 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:42,786 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:42,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:42,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:42,787 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:42,787 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:42,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:42,787 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 15:55:42,787 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1912): 4df381dfb8e2792d3aca65faf2673530/C is initiating minor compaction (all files) 2023-04-12 15:55:42,787 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:42,787 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4df381dfb8e2792d3aca65faf2673530/C in TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:42,788 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/7c0ede0be0e64184a182cb2c16de4796, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/cd6a46adcece4549aa04f48ee04f698f, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/4993656bbde34d1096a4c893bb940815] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp, totalSize=34.8 K 2023-04-12 15:55:42,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:42,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:42,788 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:42,788 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 7c0ede0be0e64184a182cb2c16de4796, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1681314941321 2023-04-12 15:55:42,789 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting cd6a46adcece4549aa04f48ee04f698f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1681314941362 2023-04-12 15:55:42,789 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 4993656bbde34d1096a4c893bb940815, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1681314941996 2023-04-12 15:55:42,792 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:42,792 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:42,792 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:42,793 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:42,793 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:42,805 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:42,805 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:42,806 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:42,806 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. started... 2023-04-12 15:55:42,806 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. as already flushing 2023-04-12 15:55:42,806 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:42,806 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:42,806 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:42,806 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:42,808 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:42,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55306 deadline: 1681315002789, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:42,809 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:42,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55336 deadline: 1681315002789, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:42,810 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:42,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55322 deadline: 1681315002790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:42,810 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] throttle.PressureAwareThroughputController(145): 4df381dfb8e2792d3aca65faf2673530#C#compaction#410 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:55:42,811 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:42,811 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:42,814 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:42,814 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:42,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:42,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:42,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:42,815 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:42,815 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:42,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:42,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:42,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:42,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:42,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:42,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:42,819 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d16f1dc[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:42,819 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:42,819 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:42,820 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:42,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:42,820 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:42,820 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:42,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:42,820 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:42,820 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:42,820 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:42,821 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:42,821 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:42,821 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:42,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:42,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:42,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:42,822 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:42,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:42,822 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:42,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55260 deadline: 1681315002807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:42,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:42,822 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:42,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55340 deadline: 1681315002809, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:42,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:42,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:42,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:42,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:42,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:42,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:42,824 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:42,824 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:42,824 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:42,824 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:42,825 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:42,825 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:42,828 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:42,828 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:42,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:42,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:42,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:42,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:42,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:42,829 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:42,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:42,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:42,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:42,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:42,829 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:42,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:42,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:42,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:42,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:42,831 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:42,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:42,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:42,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:42,833 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:42,833 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:42,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:42,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:42,834 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:42,834 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:42,834 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:42,834 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:42,834 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:42,834 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:42,834 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:42,834 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:42,834 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:42,834 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:42,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:42,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:42,834 DEBUG [Thread-3125] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:42,834 DEBUG [Thread-3125] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:42,918 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:42,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55306 deadline: 1681315002910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:42,919 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:42,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55336 deadline: 1681315002910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:42,921 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:42,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55322 deadline: 1681315002912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:42,932 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:42,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55340 deadline: 1681315002923, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:42,932 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:42,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55260 deadline: 1681315002924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:42,934 DEBUG [Thread-3125] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:42,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:42,936 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:42,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:42,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:42,937 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:42,937 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:42,937 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:42,937 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:42,938 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:42,938 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:42,939 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:42,939 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:42,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:42,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:42,939 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:42,939 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:42,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:42,940 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:42,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:42,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:42,940 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:42,940 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:42,940 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:42,941 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:42,941 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:42,941 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:42,941 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:42,941 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:42,941 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:42,941 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. started... 2023-04-12 15:55:42,941 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. as already flushing 2023-04-12 15:55:42,941 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:42,941 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:42,941 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:42,941 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:42,941 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:42,941 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:42,945 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:42,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:42,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:42,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:42,945 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:42,950 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:42,950 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:42,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:42,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:42,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:42,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:42,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:42,952 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:42,952 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d5ed73c[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:42,952 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:42,952 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:42,953 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:42,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:42,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:42,954 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:42,954 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:42,954 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:42,954 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:42,954 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:42,958 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:42,958 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:42,958 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:42,958 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:42,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:42,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:42,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:42,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:42,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:42,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:42,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:42,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:42,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:42,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:42,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:42,960 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:42,960 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:42,960 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:42,960 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:42,960 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:42,960 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:42,962 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:42,962 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:42,962 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:42,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:42,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:42,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:42,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:42,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:42,962 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:42,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:42,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:42,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:42,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:42,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:42,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:42,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:42,964 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:42,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:42,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:42,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:42,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:42,967 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:42,967 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:42,967 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:42,967 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:42,967 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:42,967 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:42,967 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:42,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:42,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:42,967 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:42,967 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:42,967 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:42,967 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:42,968 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:42,967 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:42,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:42,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:42,968 DEBUG [Thread-3125] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:42,968 DEBUG [Thread-3125] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:43,068 DEBUG [Thread-3125] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:43,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:43,070 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:43,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:43,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:43,072 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:43,072 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:43,072 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:43,072 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:43,072 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:43,072 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:43,074 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:43,074 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:43,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:43,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:43,074 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:43,074 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:43,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:43,075 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:43,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:43,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:43,075 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:43,075 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:43,075 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:43,075 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:43,075 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:43,075 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:43,076 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:43,076 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:43,076 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:43,076 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. started... 2023-04-12 15:55:43,076 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. as already flushing 2023-04-12 15:55:43,076 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:43,076 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:43,076 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:43,076 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:43,076 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:43,076 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:43,079 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:43,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:43,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:43,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:43,079 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:43,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:43,080 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:43,080 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:43,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:43,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:43,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:43,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:43,082 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:43,082 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12e20149[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:43,082 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:43,083 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:43,084 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:43,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:43,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:43,084 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:43,084 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:43,084 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:43,084 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:43,084 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:43,086 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:43,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:43,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:43,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:43,086 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:43,086 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:43,086 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:43,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:43,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:43,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:43,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:43,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:43,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:43,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:43,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:43,090 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:43,090 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:43,090 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:43,090 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:43,090 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:43,090 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:43,096 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:43,096 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:43,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:43,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:43,096 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:43,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:43,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:43,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:43,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:43,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:43,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:43,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:43,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:43,097 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:43,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:43,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:43,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:43,098 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:43,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:43,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:43,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:43,102 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:43,102 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:43,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:43,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:43,102 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:43,103 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:43,103 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:43,103 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:43,103 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:43,103 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:43,103 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:43,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:43,103 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:43,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:43,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:43,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:43,103 DEBUG [Thread-3125] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:43,103 DEBUG [Thread-3125] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:43,124 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:43,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55306 deadline: 1681315003120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:43,125 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:43,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55336 deadline: 1681315003120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:43,125 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:43,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55322 deadline: 1681315003122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:43,137 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:43,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55340 deadline: 1681315003134, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:43,137 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:43,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55260 deadline: 1681315003135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:43,199 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/A/416d50d9887248049b766b97eb1716a0 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/416d50d9887248049b766b97eb1716a0 2023-04-12 15:55:43,203 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4df381dfb8e2792d3aca65faf2673530/A of 4df381dfb8e2792d3aca65faf2673530 into 416d50d9887248049b766b97eb1716a0(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:55:43,203 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4df381dfb8e2792d3aca65faf2673530: 2023-04-12 15:55:43,203 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530., storeName=4df381dfb8e2792d3aca65faf2673530/A, priority=13, startTime=1681314942707; duration=0sec 2023-04-12 15:55:43,203 DEBUG [Thread-3125] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:43,203 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:55:43,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:43,208 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:43,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:43,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:43,210 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:43,211 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:43,211 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:43,211 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:43,211 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:43,211 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:43,212 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:43,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:43,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:43,212 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:43,213 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:43,213 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:43,214 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=68 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/A/9cd8ec3575044ec3863359f57eb1fe52 2023-04-12 15:55:43,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:43,216 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:43,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:43,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:43,217 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:43,217 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:43,217 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:43,217 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:43,217 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:43,217 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:43,217 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:43,217 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:43,218 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:43,218 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. started... 2023-04-12 15:55:43,218 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. as already flushing 2023-04-12 15:55:43,218 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:43,218 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:43,218 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:43,218 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:43,218 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:43,218 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:43,220 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:43,221 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:43,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:43,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:43,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:43,221 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:43,221 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:43,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:43,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:43,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:43,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:43,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:43,225 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:43,225 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74e10a3d[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:43,225 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:43,225 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:43,227 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:43,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:43,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:43,227 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:43,227 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:43,227 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:43,227 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:43,227 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:43,230 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:43,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:43,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:43,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:43,230 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:43,230 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:43,230 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:43,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:43,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:43,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:43,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:43,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:43,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:43,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:43,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:43,233 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:43,233 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:43,233 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:43,233 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:43,233 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:43,233 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:43,233 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=68 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/B/b5f524c594bb48a6a9f9cd6aaa313d9d 2023-04-12 15:55:43,236 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:43,236 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:43,236 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:43,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:43,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:43,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:43,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:43,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:43,237 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:43,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:43,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:43,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:43,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:43,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:43,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:43,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:43,241 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:43,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:43,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:43,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:43,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:43,243 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:43,243 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:43,243 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:43,243 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:43,243 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:43,243 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:43,244 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:43,244 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:43,244 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:43,244 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:43,244 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:43,244 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:43,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:43,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:43,244 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:43,244 DEBUG [Thread-3125] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:43,244 DEBUG [Thread-3125] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:43,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:43,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:43,271 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=68 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/C/5ee1d0c731b94394bc32f0822486bdc8 2023-04-12 15:55:43,274 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/C/5502d99bac8e4426b6b3df563d9c86a2 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/5502d99bac8e4426b6b3df563d9c86a2 2023-04-12 15:55:43,277 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/A/9cd8ec3575044ec3863359f57eb1fe52 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/9cd8ec3575044ec3863359f57eb1fe52 2023-04-12 15:55:43,281 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4df381dfb8e2792d3aca65faf2673530/C of 4df381dfb8e2792d3aca65faf2673530 into 5502d99bac8e4426b6b3df563d9c86a2(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:55:43,281 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4df381dfb8e2792d3aca65faf2673530: 2023-04-12 15:55:43,281 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530., storeName=4df381dfb8e2792d3aca65faf2673530/C, priority=13, startTime=1681314942708; duration=0sec 2023-04-12 15:55:43,281 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:55:43,282 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/9cd8ec3575044ec3863359f57eb1fe52, entries=150, sequenceid=68, filesize=11.6 K 2023-04-12 15:55:43,283 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/B/b5f524c594bb48a6a9f9cd6aaa313d9d as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/b5f524c594bb48a6a9f9cd6aaa313d9d 2023-04-12 15:55:43,286 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/b5f524c594bb48a6a9f9cd6aaa313d9d, entries=150, sequenceid=68, filesize=11.6 K 2023-04-12 15:55:43,287 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/C/5ee1d0c731b94394bc32f0822486bdc8 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/5ee1d0c731b94394bc32f0822486bdc8 2023-04-12 15:55:43,291 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/5ee1d0c731b94394bc32f0822486bdc8, entries=150, sequenceid=68, filesize=11.6 K 2023-04-12 15:55:43,292 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~69.33 KB/70990, heapSize ~194.20 KB/198864, currentSize=114.05 KB/116790 for 4df381dfb8e2792d3aca65faf2673530 in 534ms, sequenceid=68, compaction requested=false 2023-04-12 15:55:43,292 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4df381dfb8e2792d3aca65faf2673530: 2023-04-12 15:55:43,344 DEBUG [Thread-3125] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:43,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:43,347 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:43,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:43,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:43,348 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:43,348 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:43,348 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:43,348 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:43,349 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:43,349 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:43,357 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:43,357 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:43,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:43,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:43,358 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:43,358 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:43,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:43,358 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:43,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:43,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:43,359 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:43,359 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:43,359 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:43,359 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:43,359 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:43,359 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:43,359 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:43,359 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:43,359 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:43,359 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. started... 2023-04-12 15:55:43,360 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4df381dfb8e2792d3aca65faf2673530 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-12 15:55:43,360 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=A 2023-04-12 15:55:43,360 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:43,360 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=B 2023-04-12 15:55:43,360 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:43,360 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=C 2023-04-12 15:55:43,360 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:43,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on 4df381dfb8e2792d3aca65faf2673530 2023-04-12 15:55:43,429 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. as already flushing 2023-04-12 15:55:43,443 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:43,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55260 deadline: 1681315003439, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:43,444 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:43,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55336 deadline: 1681315003440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:43,445 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:43,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55322 deadline: 1681315003442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:43,448 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:43,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55306 deadline: 1681315003443, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:43,450 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:43,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55340 deadline: 1681315003444, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:43,552 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:43,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55336 deadline: 1681315003545, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:43,553 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:43,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55322 deadline: 1681315003546, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:43,554 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:43,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55306 deadline: 1681315003549, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:43,555 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:43,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55340 deadline: 1681315003551, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:43,758 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:43,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55336 deadline: 1681315003753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:43,758 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:43,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55322 deadline: 1681315003755, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:43,758 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:43,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55306 deadline: 1681315003755, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:43,758 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:43,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55340 deadline: 1681315003756, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:43,780 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=91 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/A/4f19c5585adc4da29df0032445408d3c 2023-04-12 15:55:43,800 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=91 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/B/f8d00bdf8dc34eb09fd76edfddbf4bb6 2023-04-12 15:55:43,818 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=91 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/C/327a0142253340ffa28b0bcefb7a06dd 2023-04-12 15:55:43,825 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/A/4f19c5585adc4da29df0032445408d3c as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/4f19c5585adc4da29df0032445408d3c 2023-04-12 15:55:43,829 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/4f19c5585adc4da29df0032445408d3c, entries=150, sequenceid=91, filesize=11.6 K 2023-04-12 15:55:43,830 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/B/f8d00bdf8dc34eb09fd76edfddbf4bb6 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/f8d00bdf8dc34eb09fd76edfddbf4bb6 2023-04-12 15:55:43,835 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/f8d00bdf8dc34eb09fd76edfddbf4bb6, entries=150, sequenceid=91, filesize=11.6 K 2023-04-12 15:55:43,836 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/C/327a0142253340ffa28b0bcefb7a06dd as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/327a0142253340ffa28b0bcefb7a06dd 2023-04-12 15:55:43,843 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/327a0142253340ffa28b0bcefb7a06dd, entries=150, sequenceid=91, filesize=11.6 K 2023-04-12 15:55:43,843 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 4df381dfb8e2792d3aca65faf2673530 in 484ms, sequenceid=91, compaction requested=true 2023-04-12 15:55:43,844 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4df381dfb8e2792d3aca65faf2673530: 2023-04-12 15:55:43,844 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:43,844 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:43,844 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:43,844 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:43,844 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:43,844 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:43,848 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:43,848 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:43,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:43,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:43,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:43,849 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:43,849 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:43,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:43,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:43,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:43,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:43,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:43,851 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:43,851 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e939d6c[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:43,851 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:43,851 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:43,852 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:43,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:43,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:43,852 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:43,852 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:43,852 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:43,852 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:43,852 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:43,854 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:43,854 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:43,854 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:43,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:43,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:43,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:43,854 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:43,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:43,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:43,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:43,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:43,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:43,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:43,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:43,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:43,856 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:43,856 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:43,856 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:43,856 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:43,856 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:43,856 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:43,858 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:43,858 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:43,858 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:43,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:43,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:43,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:43,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:43,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:43,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:43,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:43,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:43,859 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:43,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:43,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:43,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:43,860 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:43,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:43,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:43,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:43,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:43,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:43,863 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:43,863 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:43,863 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:43,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:43,863 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:43,863 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:43,863 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:43,863 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:43,863 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:43,863 DEBUG [Thread-3125] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:43,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:43,863 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:43,864 DEBUG [Thread-3125] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:43,864 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:43,864 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:43,864 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:43,864 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:43,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:43,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:43,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on 4df381dfb8e2792d3aca65faf2673530 2023-04-12 15:55:43,951 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4df381dfb8e2792d3aca65faf2673530 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-12 15:55:43,951 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=A 2023-04-12 15:55:43,951 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:43,951 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=B 2023-04-12 15:55:43,951 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:43,951 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=C 2023-04-12 15:55:43,951 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:43,964 DEBUG [Thread-3125] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:43,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:43,966 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:43,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:43,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:43,977 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:43,977 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:43,977 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:43,977 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:43,977 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:43,978 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:43,979 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:43,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:43,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:43,979 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:43,979 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:43,979 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:43,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:43,980 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:43,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:43,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:43,981 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:43,981 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:43,981 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:43,981 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:43,981 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:43,981 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:43,981 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:43,981 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:43,981 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. started... 2023-04-12 15:55:43,982 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. as already flushing 2023-04-12 15:55:43,982 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:43,982 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:43,982 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:43,982 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:43,983 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:43,983 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:43,984 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:43,984 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:43,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:43,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:43,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:43,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:43,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:43,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:43,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:43,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:43,986 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:43,986 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:43,986 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:43,986 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35df4eb5[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:43,986 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:43,986 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:43,989 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:43,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:43,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:43,989 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:43,989 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:43,989 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:43,989 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:43,989 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:43,990 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:43,990 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:43,991 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:43,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:43,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:43,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:43,991 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:43,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:43,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:43,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:43,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:43,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:43,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:43,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:43,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:43,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:43,993 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:43,993 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:43,993 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:43,993 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:43,994 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:43,995 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:43,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:43,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:43,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:43,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:43,995 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:43,996 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:43,996 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:43,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:43,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:43,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:44,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:44,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:44,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:44,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:44,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:44,000 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:44,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:44,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,006 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,006 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:44,006 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:44,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:44,006 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:44,006 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,006 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:44,006 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:44,006 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:44,006 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:44,006 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:44,006 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:44,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:44,006 DEBUG [Thread-3125] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:44,006 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:44,006 DEBUG [Thread-3125] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:44,006 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:44,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:44,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:44,064 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:44,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55306 deadline: 1681315004059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,066 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:44,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55260 deadline: 1681315004060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,066 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:44,066 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:44,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55340 deadline: 1681315004061, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55336 deadline: 1681315004061, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,072 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:44,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55322 deadline: 1681315004064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,106 DEBUG [Thread-3125] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:44,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:44,108 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:44,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:44,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:44,110 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:44,110 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:44,110 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:44,110 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:44,110 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:44,110 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:44,112 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,112 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:44,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:44,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:44,112 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,112 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:44,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:44,112 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:44,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:44,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:44,113 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:44,113 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:44,113 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:44,113 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:44,113 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:44,113 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:44,114 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:44,114 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:44,114 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:44,114 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. started... 2023-04-12 15:55:44,114 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. as already flushing 2023-04-12 15:55:44,114 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:44,114 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:44,114 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:44,114 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,114 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:44,114 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:44,116 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:44,116 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:44,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:44,116 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:44,116 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:44,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:44,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:44,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:44,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:44,117 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:44,117 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3735f90e[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:44,117 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:44,117 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:44,118 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:44,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:44,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:44,119 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:44,119 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:44,119 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,119 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:44,119 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:44,121 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,121 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:44,121 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:44,121 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:44,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:44,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:44,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:44,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:44,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:44,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:44,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:44,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:44,124 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:44,124 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:44,124 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:44,124 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:44,124 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:44,124 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:44,125 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:44,125 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:44,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:44,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:44,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:44,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:44,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:44,125 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:44,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:44,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:44,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:44,126 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:44,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:44,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:44,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:44,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,127 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:44,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:44,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,129 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:44,129 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:44,129 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:44,130 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:44,129 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:44,130 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:44,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:44,129 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:44,130 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:44,130 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,130 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:44,130 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:44,130 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:44,129 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:44,130 DEBUG [Thread-3125] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:44,130 DEBUG [Thread-3125] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:44,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:44,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:44,172 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:44,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55260 deadline: 1681315004167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,177 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:44,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55322 deadline: 1681315004173, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,230 DEBUG [Thread-3125] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:44,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:44,233 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:44,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:44,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:44,234 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:44,234 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:44,234 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:44,234 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:44,234 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:44,234 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:44,236 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:44,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:44,236 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:44,237 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,237 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:44,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:44,237 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:44,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:44,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:44,238 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:44,238 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:44,238 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:44,238 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:44,238 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:44,238 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:44,238 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:44,238 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:44,238 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:44,238 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. started... 2023-04-12 15:55:44,238 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. as already flushing 2023-04-12 15:55:44,239 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:44,239 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:44,239 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:44,239 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,239 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:44,239 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:44,241 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,241 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:44,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:44,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:44,242 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:44,242 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:44,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:44,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:44,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:44,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:44,243 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:44,243 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b79bd37[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:44,243 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:44,243 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:44,246 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:44,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:44,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:44,246 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:44,246 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:44,246 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:44,246 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,246 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:44,247 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:44,247 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:44,247 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,247 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:44,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:44,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:44,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:44,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:44,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:44,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:44,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:44,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:44,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:44,252 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:44,252 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:44,252 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:44,252 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:44,252 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:44,254 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:44,254 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:44,254 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:44,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:44,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:44,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:44,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:44,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:44,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:44,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:44,254 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:44,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:44,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:44,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:44,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:44,256 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:44,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:44,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,260 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,260 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:44,260 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:44,260 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:44,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:44,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:44,260 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,260 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:44,261 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:44,261 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:44,260 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:44,260 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:44,261 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:44,261 DEBUG [Thread-3125] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:44,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:44,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:44,261 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:44,261 DEBUG [Thread-3125] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:44,261 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:44,361 DEBUG [Thread-3125] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:44,362 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=104 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/A/01ef1847c2084f77a63accc1dfd5d0f7 2023-04-12 15:55:44,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:44,364 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:44,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:44,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:44,365 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:44,365 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:44,365 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:44,365 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:44,366 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:44,366 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:44,367 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:44,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:44,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:44,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:44,368 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:44,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:44,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:44,368 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:44,369 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,369 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,369 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:44,371 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:44,371 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:44,371 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:44,372 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:44,372 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:44,372 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:44,372 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:44,372 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:44,372 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. started... 2023-04-12 15:55:44,372 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. as already flushing 2023-04-12 15:55:44,372 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:44,372 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:44,372 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:44,372 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,372 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:44,372 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:44,373 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,373 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:44,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:44,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:44,374 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:44,374 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:44,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:44,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:44,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:44,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:44,377 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:44,377 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@498331c9[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:44,377 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:44,377 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:44,378 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:44,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:44,379 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=104 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/B/3dc36daffc874ce69e528b4bd10e9a69 2023-04-12 15:55:44,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:44,379 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:44,379 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:44,379 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:44,379 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,379 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:44,382 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:44,382 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:44,382 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:44,382 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:44,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:44,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:44,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:44,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:44,385 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:44,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55260 deadline: 1681315004378, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,385 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:44,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55322 deadline: 1681315004380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:44,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:44,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:44,389 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:44,389 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:44,389 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:44,389 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:44,389 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:44,389 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:44,391 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:44,391 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:44,391 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:44,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:44,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:44,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:44,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:44,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:44,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:44,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:44,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:44,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:44,393 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:44,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:44,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:44,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,394 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:44,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:44,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,396 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:44,396 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,396 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:44,396 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:44,396 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:44,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:44,396 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:44,396 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:44,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:44,396 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:44,396 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,396 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:44,396 DEBUG [Thread-3125] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:44,397 DEBUG [Thread-3125] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:44,396 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:44,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:44,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:44,397 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:44,497 DEBUG [Thread-3125] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:44,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:44,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:44,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:44,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:44,500 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:44,500 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:44,500 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:44,500 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:44,501 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:44,501 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:44,504 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,504 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:44,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:44,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:44,504 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,504 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:44,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:44,505 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:44,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:44,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:44,505 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:44,505 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:44,505 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:44,505 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:44,505 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:44,506 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:44,506 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:44,506 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:44,506 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:44,506 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. started... 2023-04-12 15:55:44,506 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. as already flushing 2023-04-12 15:55:44,506 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:44,506 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:44,506 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:44,506 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,506 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:44,506 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:44,509 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,509 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:44,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:44,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:44,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:44,511 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:44,511 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:44,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:44,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:44,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:44,512 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:44,512 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66664aa[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:44,512 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:44,512 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:44,514 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,514 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:44,514 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:44,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:44,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:44,514 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:44,514 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:44,514 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:44,515 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,515 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:44,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:44,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:44,515 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:44,515 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:44,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:44,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:44,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:44,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:44,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:44,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:44,517 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:44,517 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:44,517 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:44,518 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:44,518 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:44,518 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:44,519 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:44,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:44,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:44,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:44,519 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:44,519 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:44,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:44,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:44,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:44,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:44,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:44,524 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:44,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:44,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:44,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:44,525 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:44,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:44,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,529 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,529 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:44,529 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:44,529 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,529 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:44,529 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:44,529 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:44,530 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:44,530 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:44,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:44,530 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:44,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:44,530 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:44,530 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:44,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:44,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:44,530 DEBUG [Thread-3125] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:44,530 DEBUG [Thread-3125] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:44,572 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:44,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55336 deadline: 1681315004568, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,572 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:44,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55306 deadline: 1681315004569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,574 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:44,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55340 deadline: 1681315004571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,630 DEBUG [Thread-3125] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:44,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:44,634 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:44,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:44,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:44,635 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:44,635 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:44,635 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:44,635 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:44,636 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:44,636 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:44,637 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,637 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:44,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:44,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:44,638 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,638 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:44,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:44,638 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:44,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:44,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:44,639 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:44,639 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:44,639 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:44,639 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:44,639 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:44,639 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:44,639 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:44,639 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:44,639 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:44,639 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. started... 2023-04-12 15:55:44,639 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. as already flushing 2023-04-12 15:55:44,639 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:44,639 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:44,639 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:44,640 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,640 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:44,640 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:44,642 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:44,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:44,642 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:44,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:44,643 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:44,643 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:44,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:44,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:44,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:44,644 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:44,644 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e8300e1[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:44,644 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:44,644 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:44,645 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:44,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:44,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:44,645 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:44,645 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:44,645 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:44,645 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,646 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:44,647 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:44,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:44,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:44,648 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:44,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:44,648 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:44,648 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:44,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:44,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:44,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:44,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:44,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:44,650 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:44,650 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:44,650 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:44,650 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:44,650 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:44,652 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:44,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:44,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:44,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:44,653 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:44,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:44,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:44,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:44,653 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,653 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:44,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:44,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:44,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:44,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:44,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:44,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:44,656 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:44,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:44,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,658 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,658 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:44,658 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:44,658 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,658 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:44,658 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:44,658 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:44,658 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:44,658 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:44,658 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:44,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:44,660 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:44,658 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:44,660 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:44,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:44,660 DEBUG [Thread-3125] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:44,660 DEBUG [Thread-3125] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:44,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:44,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:44,688 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:44,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55322 deadline: 1681315004686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,694 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:44,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55260 deadline: 1681315004688, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,760 DEBUG [Thread-3125] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:44,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:44,762 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:44,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:44,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:44,764 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:44,764 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:44,764 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:44,764 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:44,764 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:44,764 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:44,766 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:44,766 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:44,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:44,766 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,766 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:44,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:44,766 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:44,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:44,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:44,767 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:44,767 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:44,767 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:44,767 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:44,767 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:44,767 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:44,767 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:44,767 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:44,768 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:44,768 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. started... 2023-04-12 15:55:44,768 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. as already flushing 2023-04-12 15:55:44,768 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:44,768 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:44,768 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:44,768 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,768 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:44,768 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:44,771 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:44,771 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:44,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:44,771 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:44,771 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:44,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:44,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:44,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:44,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:44,773 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:44,773 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18958603[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:44,773 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:44,773 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:44,774 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:44,774 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,774 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:44,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:44,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:44,775 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:44,775 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:44,775 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:44,776 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:44,776 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:44,776 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,776 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:44,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:44,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:44,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:44,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:44,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:44,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:44,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:44,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:44,780 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:44,780 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:44,781 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:44,781 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:44,781 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:44,781 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:44,782 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:44,782 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:44,782 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:44,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:44,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:44,783 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:44,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:44,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:44,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:44,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:44,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:44,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:44,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:44,784 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:44,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:44,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:44,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:44,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,789 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:44,789 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:44,789 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:44,789 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:44,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:44,789 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:44,789 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:44,789 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:44,789 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:44,789 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:44,789 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,790 DEBUG [Thread-3125] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:44,790 DEBUG [Thread-3125] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:44,790 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:44,790 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:44,790 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:44,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:44,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:44,802 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=104 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/C/6a2dbc477d024c35ba743529d4cb5e3d 2023-04-12 15:55:44,806 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/A/01ef1847c2084f77a63accc1dfd5d0f7 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/01ef1847c2084f77a63accc1dfd5d0f7 2023-04-12 15:55:44,810 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/01ef1847c2084f77a63accc1dfd5d0f7, entries=150, sequenceid=104, filesize=11.6 K 2023-04-12 15:55:44,810 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/B/3dc36daffc874ce69e528b4bd10e9a69 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/3dc36daffc874ce69e528b4bd10e9a69 2023-04-12 15:55:44,817 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/3dc36daffc874ce69e528b4bd10e9a69, entries=100, sequenceid=104, filesize=9.3 K 2023-04-12 15:55:44,817 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/C/6a2dbc477d024c35ba743529d4cb5e3d as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/6a2dbc477d024c35ba743529d4cb5e3d 2023-04-12 15:55:44,823 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/6a2dbc477d024c35ba743529d4cb5e3d, entries=100, sequenceid=104, filesize=9.3 K 2023-04-12 15:55:44,824 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 4df381dfb8e2792d3aca65faf2673530 in 873ms, sequenceid=104, compaction requested=true 2023-04-12 15:55:44,824 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4df381dfb8e2792d3aca65faf2673530: 2023-04-12 15:55:44,824 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:55:44,824 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-12 15:55:44,824 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-12 15:55:44,824 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:55:44,825 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:55:44,827 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45303 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-12 15:55:44,827 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1912): 4df381dfb8e2792d3aca65faf2673530/B is initiating minor compaction (all files) 2023-04-12 15:55:44,827 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47643 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-12 15:55:44,827 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4df381dfb8e2792d3aca65faf2673530/B in TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:44,827 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1912): 4df381dfb8e2792d3aca65faf2673530/A is initiating minor compaction (all files) 2023-04-12 15:55:44,828 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/7a421934a8794a71a2e53fc012068644, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/b5f524c594bb48a6a9f9cd6aaa313d9d, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/f8d00bdf8dc34eb09fd76edfddbf4bb6, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/3dc36daffc874ce69e528b4bd10e9a69] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp, totalSize=44.2 K 2023-04-12 15:55:44,828 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4df381dfb8e2792d3aca65faf2673530/A in TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:44,828 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/416d50d9887248049b766b97eb1716a0, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/9cd8ec3575044ec3863359f57eb1fe52, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/4f19c5585adc4da29df0032445408d3c, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/01ef1847c2084f77a63accc1dfd5d0f7] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp, totalSize=46.5 K 2023-04-12 15:55:44,828 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 7a421934a8794a71a2e53fc012068644, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1681314941996 2023-04-12 15:55:44,828 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 416d50d9887248049b766b97eb1716a0, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1681314941996 2023-04-12 15:55:44,828 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting b5f524c594bb48a6a9f9cd6aaa313d9d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1681314942645 2023-04-12 15:55:44,828 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 9cd8ec3575044ec3863359f57eb1fe52, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1681314942645 2023-04-12 15:55:44,829 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting f8d00bdf8dc34eb09fd76edfddbf4bb6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=91, earliestPutTs=1681314942789 2023-04-12 15:55:44,829 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 4f19c5585adc4da29df0032445408d3c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=91, earliestPutTs=1681314942789 2023-04-12 15:55:44,829 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 3dc36daffc874ce69e528b4bd10e9a69, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=104, earliestPutTs=1681314943442 2023-04-12 15:55:44,830 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 01ef1847c2084f77a63accc1dfd5d0f7, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=104, earliestPutTs=1681314943442 2023-04-12 15:55:44,838 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] throttle.PressureAwareThroughputController(145): 4df381dfb8e2792d3aca65faf2673530#B#compaction#419 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:55:44,842 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4df381dfb8e2792d3aca65faf2673530#A#compaction#420 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:55:44,868 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/A/38c3d06ff4414f5aa00fdcaf10b496c9 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/38c3d06ff4414f5aa00fdcaf10b496c9 2023-04-12 15:55:44,868 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/B/6a51ebeee7ef4460acf36a60e5e543ab as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/6a51ebeee7ef4460acf36a60e5e543ab 2023-04-12 15:55:44,873 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 4df381dfb8e2792d3aca65faf2673530/A of 4df381dfb8e2792d3aca65faf2673530 into 38c3d06ff4414f5aa00fdcaf10b496c9(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:55:44,873 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4df381dfb8e2792d3aca65faf2673530: 2023-04-12 15:55:44,873 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530., storeName=4df381dfb8e2792d3aca65faf2673530/A, priority=12, startTime=1681314944824; duration=0sec 2023-04-12 15:55:44,873 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:55:44,873 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-12 15:55:44,874 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 4df381dfb8e2792d3aca65faf2673530/B of 4df381dfb8e2792d3aca65faf2673530 into 6a51ebeee7ef4460acf36a60e5e543ab(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:55:44,874 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4df381dfb8e2792d3aca65faf2673530: 2023-04-12 15:55:44,874 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530., storeName=4df381dfb8e2792d3aca65faf2673530/B, priority=12, startTime=1681314944824; duration=0sec 2023-04-12 15:55:44,874 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:55:44,876 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45303 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-12 15:55:44,876 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1912): 4df381dfb8e2792d3aca65faf2673530/C is initiating minor compaction (all files) 2023-04-12 15:55:44,876 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4df381dfb8e2792d3aca65faf2673530/C in TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:44,876 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/5502d99bac8e4426b6b3df563d9c86a2, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/5ee1d0c731b94394bc32f0822486bdc8, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/327a0142253340ffa28b0bcefb7a06dd, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/6a2dbc477d024c35ba743529d4cb5e3d] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp, totalSize=44.2 K 2023-04-12 15:55:44,877 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 5502d99bac8e4426b6b3df563d9c86a2, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1681314941996 2023-04-12 15:55:44,877 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 5ee1d0c731b94394bc32f0822486bdc8, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1681314942645 2023-04-12 15:55:44,877 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 327a0142253340ffa28b0bcefb7a06dd, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=91, earliestPutTs=1681314942789 2023-04-12 15:55:44,877 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 6a2dbc477d024c35ba743529d4cb5e3d, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=104, earliestPutTs=1681314943442 2023-04-12 15:55:44,884 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4df381dfb8e2792d3aca65faf2673530#C#compaction#421 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:55:44,890 DEBUG [Thread-3125] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:44,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:44,892 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:44,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:44,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:44,893 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:44,893 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:44,893 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:44,893 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:44,894 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:44,894 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:44,894 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/C/0953ddd414e94905b98b32b88c8091fc as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/0953ddd414e94905b98b32b88c8091fc 2023-04-12 15:55:44,895 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,895 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:44,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:44,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:44,895 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:44,895 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:44,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:44,896 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:44,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:44,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:44,896 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:44,897 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:44,897 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:44,897 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:44,897 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:44,897 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:44,897 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:44,897 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:44,897 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:44,897 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. started... 2023-04-12 15:55:44,897 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4df381dfb8e2792d3aca65faf2673530 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-12 15:55:44,897 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=A 2023-04-12 15:55:44,897 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:44,898 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=B 2023-04-12 15:55:44,898 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:44,898 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=C 2023-04-12 15:55:44,898 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:44,900 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 4df381dfb8e2792d3aca65faf2673530/C of 4df381dfb8e2792d3aca65faf2673530 into 0953ddd414e94905b98b32b88c8091fc(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:55:44,900 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4df381dfb8e2792d3aca65faf2673530: 2023-04-12 15:55:44,900 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530., storeName=4df381dfb8e2792d3aca65faf2673530/C, priority=12, startTime=1681314944825; duration=0sec 2023-04-12 15:55:44,901 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:55:44,924 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-12 15:55:44,947 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=125 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/A/a42b5fd96b5041358fe28898cc3ea568 2023-04-12 15:55:44,989 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=125 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/B/3923040399a146cdbbe12bdac999a54c 2023-04-12 15:55:45,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on 4df381dfb8e2792d3aca65faf2673530 2023-04-12 15:55:45,195 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. as already flushing 2023-04-12 15:55:45,214 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:45,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55322 deadline: 1681315005213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:45,217 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:45,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55260 deadline: 1681315005214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:45,319 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:45,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55322 deadline: 1681315005315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:45,321 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:45,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55260 deadline: 1681315005318, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:45,403 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=125 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/C/6065b13feb0c461db9e3bf535ef2c109 2023-04-12 15:55:45,407 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/A/a42b5fd96b5041358fe28898cc3ea568 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/a42b5fd96b5041358fe28898cc3ea568 2023-04-12 15:55:45,411 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/a42b5fd96b5041358fe28898cc3ea568, entries=150, sequenceid=125, filesize=11.6 K 2023-04-12 15:55:45,412 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/B/3923040399a146cdbbe12bdac999a54c as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/3923040399a146cdbbe12bdac999a54c 2023-04-12 15:55:45,415 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/3923040399a146cdbbe12bdac999a54c, entries=150, sequenceid=125, filesize=11.6 K 2023-04-12 15:55:45,420 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/C/6065b13feb0c461db9e3bf535ef2c109 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/6065b13feb0c461db9e3bf535ef2c109 2023-04-12 15:55:45,425 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/6065b13feb0c461db9e3bf535ef2c109, entries=150, sequenceid=125, filesize=11.6 K 2023-04-12 15:55:45,425 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 4df381dfb8e2792d3aca65faf2673530 in 528ms, sequenceid=125, compaction requested=false 2023-04-12 15:55:45,427 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4df381dfb8e2792d3aca65faf2673530: 2023-04-12 15:55:45,429 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:45,429 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:45,429 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:45,429 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:45,429 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:45,429 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:45,434 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:45,434 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:45,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:45,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:45,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:45,434 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:45,434 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:45,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:45,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:45,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:45,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:45,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:45,436 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:45,436 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4848f1a1[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:45,436 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:45,436 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:45,437 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:45,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:45,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:45,437 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:45,437 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:45,437 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:45,439 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:45,439 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:45,440 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:45,440 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:45,440 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:45,440 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:45,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:45,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:45,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:45,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:45,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:45,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:45,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:45,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:45,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:45,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:45,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:45,442 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:45,442 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:45,442 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:45,442 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:45,442 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:45,442 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:45,444 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:45,444 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:45,444 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:45,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:45,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:45,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:45,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:45,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:45,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:45,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:45,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:45,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:45,445 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:45,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:45,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:45,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:45,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:45,446 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:45,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:45,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:45,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:45,448 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:45,448 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:45,448 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:45,448 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:45,448 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:45,448 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:45,448 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:45,449 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:45,449 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:45,449 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:45,449 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:45,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:45,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:45,449 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:45,449 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:45,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:45,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:45,449 DEBUG [Thread-3125] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:45,449 DEBUG [Thread-3125] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:45,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on 4df381dfb8e2792d3aca65faf2673530 2023-04-12 15:55:45,523 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4df381dfb8e2792d3aca65faf2673530 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-12 15:55:45,523 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=A 2023-04-12 15:55:45,523 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:45,523 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=B 2023-04-12 15:55:45,523 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:45,523 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=C 2023-04-12 15:55:45,523 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:45,537 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=140 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/A/ac2f5e8ce9a24bd1bdd0c04af1515f82 2023-04-12 15:55:45,549 DEBUG [Thread-3125] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:45,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:45,552 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:45,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:45,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:45,553 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:45,553 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:45,553 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:45,553 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:45,553 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:45,554 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:45,555 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:45,555 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:45,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:45,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:45,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:45,555 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:45,555 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:45,555 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:45,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:45,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:45,556 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:45,556 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:45,556 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:45,556 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:45,556 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:45,556 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:45,556 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:45,557 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:45,557 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:45,557 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. started... 2023-04-12 15:55:45,557 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. as already flushing 2023-04-12 15:55:45,557 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:45,557 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:45,557 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:45,557 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:45,557 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:45,557 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:45,561 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:45,561 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:45,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:45,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:45,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:45,561 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:45,561 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:45,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:45,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:45,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:45,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:45,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:45,563 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:45,563 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7dbe9f36[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:45,563 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:45,563 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:45,564 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:45,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:45,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:45,564 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:45,564 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:45,564 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:45,564 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:45,565 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:45,566 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:45,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:45,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:45,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:45,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:45,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:45,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:45,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:45,567 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:45,567 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:45,567 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:45,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:45,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:45,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:45,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:45,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:45,569 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:45,569 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:45,569 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:45,569 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:45,569 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:45,571 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:45,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:45,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:45,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:45,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:45,572 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:45,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:45,572 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:45,572 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:45,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:45,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:45,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:45,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:45,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:45,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:45,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:45,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:45,574 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:45,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:45,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:45,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:45,578 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:45,578 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:45,578 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:45,578 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:45,578 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:45,578 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:45,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:45,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:45,578 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:45,578 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:45,578 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:45,578 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:45,578 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:45,578 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:45,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:45,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:45,578 DEBUG [Thread-3125] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:45,579 DEBUG [Thread-3125] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:45,585 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:45,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55336 deadline: 1681315005581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:45,591 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:45,591 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:45,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55260 deadline: 1681315005582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:45,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55306 deadline: 1681315005583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:45,594 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:45,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55322 deadline: 1681315005585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:45,595 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:45,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55340 deadline: 1681315005585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:45,679 DEBUG [Thread-3125] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:45,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:45,681 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:45,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:45,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:45,682 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:45,682 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:45,682 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:45,682 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:45,683 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:45,683 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:45,684 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:45,684 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:45,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:45,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:45,684 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:45,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:45,684 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:45,685 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:45,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:45,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:45,685 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:45,685 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:45,685 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:45,685 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:45,685 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:45,685 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:45,686 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:45,686 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:45,686 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:45,686 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. started... 2023-04-12 15:55:45,687 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. as already flushing 2023-04-12 15:55:45,687 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:45,687 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:45,687 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:45,687 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:45,687 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:45,687 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:45,688 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:45,688 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:45,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:45,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:45,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:45,688 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:45,688 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:45,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:45,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:45,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:45,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:45,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:45,690 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:45,690 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:45,690 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7bbc1c68[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:45,690 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:45,691 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:45,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:45,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:45,691 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:45,691 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:45,692 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:45,692 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:45,692 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:45,694 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:45,694 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:45,694 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:45,694 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:45,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:45,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:45,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:45,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:45,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:45,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:45,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:45,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:45,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:45,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:45,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:45,697 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:45,697 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:45,697 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:45,697 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:45,697 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:45,697 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:45,699 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:45,699 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:45,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:45,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:45,699 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:45,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:45,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:45,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:45,699 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:45,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55306 deadline: 1681315005693, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:45,700 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:45,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55260 deadline: 1681315005693, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:45,700 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:45,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55322 deadline: 1681315005695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:45,700 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:45,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55340 deadline: 1681315005696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:45,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:45,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:45,700 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:45,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:45,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:45,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:45,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:45,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:45,702 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:45,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:45,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:45,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:45,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:45,704 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:45,704 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:45,704 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:45,704 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:45,704 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:45,704 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:45,704 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:45,704 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:45,704 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:45,704 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:45,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:45,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:45,704 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:45,704 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:45,705 DEBUG [Thread-3125] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:45,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:45,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:45,705 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:45,705 DEBUG [Thread-3125] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:45,805 DEBUG [Thread-3125] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:45,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:45,808 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:45,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:45,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:45,809 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:45,809 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:45,809 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:45,809 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:45,809 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:45,810 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:45,811 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:45,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:45,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:45,811 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:45,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:45,811 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:45,812 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:45,812 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:45,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:45,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:45,812 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:45,812 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:45,812 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:45,812 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:45,812 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:45,812 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:45,813 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:45,813 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:45,813 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:45,813 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. started... 2023-04-12 15:55:45,813 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. as already flushing 2023-04-12 15:55:45,813 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:45,813 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:45,813 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:45,813 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:45,813 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:45,813 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:45,815 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:45,816 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:45,816 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:45,816 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:45,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:45,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:45,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:45,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:45,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:45,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:45,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:45,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:45,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:45,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4fad3882[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:45,818 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:45,818 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:45,819 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:45,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:45,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:45,819 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:45,819 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:45,819 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:45,824 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:45,824 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:45,824 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:45,824 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:45,824 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:45,824 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:45,824 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:45,824 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:45,824 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:45,824 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:45,825 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:45,827 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:45,827 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:45,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:45,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:45,827 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:45,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:45,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:45,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:45,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:45,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:45,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:45,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:45,829 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:45,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:45,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:45,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:45,830 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:45,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:45,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:45,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:45,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:45,834 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:45,834 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:45,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:45,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:45,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:45,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:45,835 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:45,836 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:45,836 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:45,836 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:45,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:45,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:45,836 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:45,836 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:45,836 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:45,836 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:45,836 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:45,836 DEBUG [Thread-3125] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:45,836 DEBUG [Thread-3125] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:45,902 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:45,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55306 deadline: 1681315005901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:45,902 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:45,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55340 deadline: 1681315005901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:45,903 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:45,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55260 deadline: 1681315005902, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:45,906 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:45,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55322 deadline: 1681315005902, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:45,936 DEBUG [Thread-3125] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:45,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:45,938 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:45,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:45,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:45,939 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:45,939 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:45,939 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:45,939 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:45,940 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:45,940 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:45,941 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:45,941 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:45,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:45,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:45,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:45,941 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:45,941 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:45,942 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:45,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:45,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:45,942 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:45,942 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:45,942 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:45,942 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:45,942 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:45,942 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:45,942 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:45,942 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:45,942 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:45,943 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. started... 2023-04-12 15:55:45,943 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. as already flushing 2023-04-12 15:55:45,943 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:45,943 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:45,943 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:45,943 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:45,943 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:45,943 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:45,944 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:45,944 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:45,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:45,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:45,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:45,944 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:45,945 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:45,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:45,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:45,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:45,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:45,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:45,946 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:45,946 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40e70d2f[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:45,946 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:45,946 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:45,948 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:45,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:45,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:45,948 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:45,948 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:45,948 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:45,948 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:45,948 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:45,949 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:45,949 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:45,949 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:45,949 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:45,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:45,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:45,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:45,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:45,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:45,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:45,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:45,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:45,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:45,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:45,951 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=140 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/B/74e22206aba34b0196a6326ba5a9630b 2023-04-12 15:55:45,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:45,952 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:45,952 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:45,952 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:45,952 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:45,952 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:45,952 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:45,954 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:45,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:45,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:45,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:45,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:45,954 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:45,954 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:45,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:45,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:45,955 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:45,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:45,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:45,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:45,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:45,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:45,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:45,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:45,956 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:45,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:45,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:45,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:45,959 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:45,959 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:45,959 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:45,959 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:45,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:45,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:45,959 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:45,959 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:45,959 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:45,959 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:45,959 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:45,959 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:45,959 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:45,959 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:45,959 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:45,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:45,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:45,959 DEBUG [Thread-3125] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:45,960 DEBUG [Thread-3125] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:45,963 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=140 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/C/0b8b3872afca4544b5a97ec8f8af02c0 2023-04-12 15:55:45,966 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/A/ac2f5e8ce9a24bd1bdd0c04af1515f82 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/ac2f5e8ce9a24bd1bdd0c04af1515f82 2023-04-12 15:55:45,969 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/ac2f5e8ce9a24bd1bdd0c04af1515f82, entries=200, sequenceid=140, filesize=14.1 K 2023-04-12 15:55:45,970 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/B/74e22206aba34b0196a6326ba5a9630b as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/74e22206aba34b0196a6326ba5a9630b 2023-04-12 15:55:45,974 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/74e22206aba34b0196a6326ba5a9630b, entries=150, sequenceid=140, filesize=11.8 K 2023-04-12 15:55:45,975 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/C/0b8b3872afca4544b5a97ec8f8af02c0 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/0b8b3872afca4544b5a97ec8f8af02c0 2023-04-12 15:55:45,978 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/0b8b3872afca4544b5a97ec8f8af02c0, entries=150, sequenceid=140, filesize=11.8 K 2023-04-12 15:55:45,978 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 4df381dfb8e2792d3aca65faf2673530 in 455ms, sequenceid=140, compaction requested=true 2023-04-12 15:55:45,978 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4df381dfb8e2792d3aca65faf2673530: 2023-04-12 15:55:45,978 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:55:45,978 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:55:45,978 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 15:55:45,978 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 15:55:45,978 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:55:45,980 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38438 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 15:55:45,980 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1912): 4df381dfb8e2792d3aca65faf2673530/A is initiating minor compaction (all files) 2023-04-12 15:55:45,980 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4df381dfb8e2792d3aca65faf2673530/A in TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:45,980 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/38c3d06ff4414f5aa00fdcaf10b496c9, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/a42b5fd96b5041358fe28898cc3ea568, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/ac2f5e8ce9a24bd1bdd0c04af1515f82] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp, totalSize=37.5 K 2023-04-12 15:55:45,980 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36048 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 15:55:45,980 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1912): 4df381dfb8e2792d3aca65faf2673530/B is initiating minor compaction (all files) 2023-04-12 15:55:45,980 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4df381dfb8e2792d3aca65faf2673530/B in TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:45,980 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/6a51ebeee7ef4460acf36a60e5e543ab, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/3923040399a146cdbbe12bdac999a54c, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/74e22206aba34b0196a6326ba5a9630b] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp, totalSize=35.2 K 2023-04-12 15:55:45,981 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 38c3d06ff4414f5aa00fdcaf10b496c9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=104, earliestPutTs=1681314942805 2023-04-12 15:55:45,982 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 6a51ebeee7ef4460acf36a60e5e543ab, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=104, earliestPutTs=1681314942805 2023-04-12 15:55:45,982 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting a42b5fd96b5041358fe28898cc3ea568, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=125, earliestPutTs=1681314944035 2023-04-12 15:55:45,982 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 3923040399a146cdbbe12bdac999a54c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=125, earliestPutTs=1681314944035 2023-04-12 15:55:45,982 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting ac2f5e8ce9a24bd1bdd0c04af1515f82, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=140, earliestPutTs=1681314945194 2023-04-12 15:55:45,982 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 74e22206aba34b0196a6326ba5a9630b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=140, earliestPutTs=1681314945194 2023-04-12 15:55:45,989 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4df381dfb8e2792d3aca65faf2673530#A#compaction#428 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:55:45,989 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] throttle.PressureAwareThroughputController(145): 4df381dfb8e2792d3aca65faf2673530#B#compaction#429 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:55:46,009 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/A/b01e45e87f434bf1bb0769639a1bbda6 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/b01e45e87f434bf1bb0769639a1bbda6 2023-04-12 15:55:46,020 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4df381dfb8e2792d3aca65faf2673530/A of 4df381dfb8e2792d3aca65faf2673530 into b01e45e87f434bf1bb0769639a1bbda6(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:55:46,020 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4df381dfb8e2792d3aca65faf2673530: 2023-04-12 15:55:46,020 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530., storeName=4df381dfb8e2792d3aca65faf2673530/A, priority=13, startTime=1681314945978; duration=0sec 2023-04-12 15:55:46,020 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:55:46,020 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 15:55:46,021 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36048 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 15:55:46,021 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1912): 4df381dfb8e2792d3aca65faf2673530/C is initiating minor compaction (all files) 2023-04-12 15:55:46,021 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4df381dfb8e2792d3aca65faf2673530/C in TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:46,021 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/0953ddd414e94905b98b32b88c8091fc, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/6065b13feb0c461db9e3bf535ef2c109, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/0b8b3872afca4544b5a97ec8f8af02c0] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp, totalSize=35.2 K 2023-04-12 15:55:46,022 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 0953ddd414e94905b98b32b88c8091fc, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=104, earliestPutTs=1681314942805 2023-04-12 15:55:46,022 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 6065b13feb0c461db9e3bf535ef2c109, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=125, earliestPutTs=1681314944035 2023-04-12 15:55:46,022 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 0b8b3872afca4544b5a97ec8f8af02c0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=140, earliestPutTs=1681314945194 2023-04-12 15:55:46,027 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4df381dfb8e2792d3aca65faf2673530#C#compaction#430 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:55:46,060 DEBUG [Thread-3125] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:46,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:46,061 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:46,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:46,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:46,063 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:46,063 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:46,063 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:46,063 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:46,063 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:46,063 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:46,065 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:46,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:46,065 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:46,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:46,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:46,065 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:46,066 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:46,066 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:46,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:46,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:46,068 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:46,068 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:46,068 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:46,068 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:46,068 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:46,068 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:46,068 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:46,068 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:46,068 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:46,068 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. started... 2023-04-12 15:55:46,069 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4df381dfb8e2792d3aca65faf2673530 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-12 15:55:46,069 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=A 2023-04-12 15:55:46,069 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:46,069 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=B 2023-04-12 15:55:46,069 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:46,069 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=C 2023-04-12 15:55:46,069 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:46,081 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=159 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/A/826fceaef48449bf93a2341960586ea9 2023-04-12 15:55:46,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on 4df381dfb8e2792d3aca65faf2673530 2023-04-12 15:55:46,209 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. as already flushing 2023-04-12 15:55:46,225 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:46,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55306 deadline: 1681315006219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:46,227 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:46,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55340 deadline: 1681315006224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:46,230 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:46,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55260 deadline: 1681315006225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:46,231 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:46,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55322 deadline: 1681315006226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:46,329 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:46,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55306 deadline: 1681315006326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:46,332 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:46,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55340 deadline: 1681315006328, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:46,336 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:46,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55260 deadline: 1681315006331, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:46,336 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:46,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55322 deadline: 1681315006332, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:46,413 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/B/1b2b9160212044558619e4e1f26dcd66 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/1b2b9160212044558619e4e1f26dcd66 2023-04-12 15:55:46,417 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4df381dfb8e2792d3aca65faf2673530/B of 4df381dfb8e2792d3aca65faf2673530 into 1b2b9160212044558619e4e1f26dcd66(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:55:46,417 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4df381dfb8e2792d3aca65faf2673530: 2023-04-12 15:55:46,417 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530., storeName=4df381dfb8e2792d3aca65faf2673530/B, priority=13, startTime=1681314945978; duration=0sec 2023-04-12 15:55:46,417 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:55:46,438 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/C/5b7ec464d1624b718927e56e7170276e as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/5b7ec464d1624b718927e56e7170276e 2023-04-12 15:55:46,442 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4df381dfb8e2792d3aca65faf2673530/C of 4df381dfb8e2792d3aca65faf2673530 into 5b7ec464d1624b718927e56e7170276e(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:55:46,442 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4df381dfb8e2792d3aca65faf2673530: 2023-04-12 15:55:46,442 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530., storeName=4df381dfb8e2792d3aca65faf2673530/C, priority=13, startTime=1681314945978; duration=0sec 2023-04-12 15:55:46,442 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:55:46,497 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=159 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/B/870e6f9636cc4d209e233b8b7e62e3ab 2023-04-12 15:55:46,507 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=159 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/C/415c0b98489944ebb2e83e4e1e67bc33 2023-04-12 15:55:46,511 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/A/826fceaef48449bf93a2341960586ea9 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/826fceaef48449bf93a2341960586ea9 2023-04-12 15:55:46,515 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/826fceaef48449bf93a2341960586ea9, entries=150, sequenceid=159, filesize=11.8 K 2023-04-12 15:55:46,516 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/B/870e6f9636cc4d209e233b8b7e62e3ab as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/870e6f9636cc4d209e233b8b7e62e3ab 2023-04-12 15:55:46,519 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/870e6f9636cc4d209e233b8b7e62e3ab, entries=150, sequenceid=159, filesize=11.8 K 2023-04-12 15:55:46,520 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/C/415c0b98489944ebb2e83e4e1e67bc33 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/415c0b98489944ebb2e83e4e1e67bc33 2023-04-12 15:55:46,527 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/415c0b98489944ebb2e83e4e1e67bc33, entries=150, sequenceid=159, filesize=11.8 K 2023-04-12 15:55:46,528 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=80.51 KB/82440 for 4df381dfb8e2792d3aca65faf2673530 in 460ms, sequenceid=159, compaction requested=false 2023-04-12 15:55:46,528 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4df381dfb8e2792d3aca65faf2673530: 2023-04-12 15:55:46,528 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:46,528 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:46,528 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:46,528 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:46,528 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:46,528 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:46,530 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:46,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:46,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:46,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:46,530 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:46,530 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:46,530 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:46,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:46,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:46,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:46,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:46,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:46,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:46,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49297d2a[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:46,532 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:46,532 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:46,534 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:46,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:46,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:46,534 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:46,534 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:46,534 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:46,534 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:46,534 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:46,535 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:46,535 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:46,535 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:46,535 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:46,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:46,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:46,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:46,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:46,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:46,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:46,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:46,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:46,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:46,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:46,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:46,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:46,538 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:46,538 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:46,538 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:46,538 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:46,538 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:46,541 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:46,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on 4df381dfb8e2792d3aca65faf2673530 2023-04-12 15:55:46,541 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4df381dfb8e2792d3aca65faf2673530 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-12 15:55:46,541 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:46,541 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:46,541 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=A 2023-04-12 15:55:46,541 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:46,541 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=B 2023-04-12 15:55:46,541 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:46,541 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=C 2023-04-12 15:55:46,541 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:46,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:46,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:46,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:46,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:46,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:46,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:46,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:46,542 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:46,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:46,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:46,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:46,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:46,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:46,544 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:46,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:46,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:46,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:46,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:46,546 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:46,546 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:46,546 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:46,546 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:46,546 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:46,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:46,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:46,546 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:46,547 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:46,547 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:46,547 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:46,547 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:46,547 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:46,547 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:46,547 DEBUG [Thread-3125] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:46,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:46,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:46,547 DEBUG [Thread-3125] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:46,547 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:46,556 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=178 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/A/1ea3d3930ba44e338b3ba505142519ed 2023-04-12 15:55:46,565 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:46,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55322 deadline: 1681315006558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:46,571 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:46,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55340 deadline: 1681315006563, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:46,575 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:46,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55260 deadline: 1681315006563, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:46,575 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:46,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55306 deadline: 1681315006564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:46,647 DEBUG [Thread-3125] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:46,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:46,649 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:46,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:46,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:46,650 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:46,650 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:46,650 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:46,650 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:46,651 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:46,651 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:46,652 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:46,652 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:46,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:46,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:46,653 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:46,653 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:46,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:46,653 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:46,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:46,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:46,653 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:46,654 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:46,654 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:46,654 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:46,654 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:46,654 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:46,654 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:46,654 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:46,654 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:46,654 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. started... 2023-04-12 15:55:46,654 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. as already flushing 2023-04-12 15:55:46,654 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:46,654 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:46,654 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:46,654 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:46,654 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:46,654 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:46,656 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:46,656 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:46,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:46,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:46,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:46,656 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:46,656 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:46,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:46,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:46,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:46,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:46,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:46,658 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:46,658 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@526d773b[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:46,658 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:46,659 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:46,660 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:46,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:46,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:46,660 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:46,660 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:46,660 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:46,660 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:46,660 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:46,662 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:46,662 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:46,662 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:46,662 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:46,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:46,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:46,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:46,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:46,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:46,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:46,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:46,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:46,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:46,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:46,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:46,664 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:46,664 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:46,664 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:46,664 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:46,664 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:46,664 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:46,668 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:46,668 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:46,668 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:46,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:46,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:46,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:46,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:46,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:46,669 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:46,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:46,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:46,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:46,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:46,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:46,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:46,670 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:46,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:46,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:46,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:46,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:46,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:46,674 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:46,674 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:46,674 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:46,674 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:46,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:46,674 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:46,674 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:46,674 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:46,674 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:46,674 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:46,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:46,674 DEBUG [Thread-3125] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:46,674 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:46,675 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:46,675 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:46,675 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:46,675 DEBUG [Thread-3125] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:46,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:46,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:46,675 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:46,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55322 deadline: 1681315006670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:46,680 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:46,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55340 deadline: 1681315006672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:46,680 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:46,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55260 deadline: 1681315006676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:46,680 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:46,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55306 deadline: 1681315006676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:46,775 DEBUG [Thread-3125] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:46,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:46,777 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:46,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:46,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:46,778 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:46,779 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:46,779 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:46,779 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:46,780 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:46,780 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:46,782 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:46,782 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:46,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:46,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:46,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:46,782 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:46,782 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:46,783 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:46,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:46,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:46,783 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:46,783 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:46,783 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:46,783 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:46,783 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:46,784 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:46,784 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:46,784 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:46,784 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. started... 2023-04-12 15:55:46,784 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. as already flushing 2023-04-12 15:55:46,784 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:46,784 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:46,784 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:46,784 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:46,784 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:46,784 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:46,786 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:46,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:46,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:46,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:46,786 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:46,786 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:46,786 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:46,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:46,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:46,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:46,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:46,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:46,788 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:46,788 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74223cdc[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:46,788 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:46,788 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:46,790 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:46,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:46,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:46,790 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:46,790 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:46,790 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:46,791 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:46,791 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:46,792 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:46,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:46,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:46,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:46,792 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:46,792 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:46,793 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:46,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:46,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:46,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:46,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:46,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:46,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:46,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:46,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:46,795 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:46,795 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:46,795 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:46,795 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:46,795 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:46,795 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:46,797 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:46,797 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:46,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:46,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:46,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:46,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:46,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:46,797 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:46,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:46,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:46,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:46,798 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:46,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:46,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:46,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:46,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:46,799 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:46,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:46,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:46,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:46,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:46,801 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:46,801 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:46,801 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:46,801 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:46,801 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:46,802 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:46,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:46,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:46,801 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:46,802 DEBUG [Thread-3125] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:46,802 DEBUG [Thread-3125] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:46,801 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:46,802 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:46,802 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:46,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:46,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:46,802 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:46,802 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:46,802 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:46,881 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:46,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55322 deadline: 1681315006877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:46,885 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:46,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55340 deadline: 1681315006881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:46,885 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:46,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55260 deadline: 1681315006881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:46,886 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:46,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55306 deadline: 1681315006882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:46,902 DEBUG [Thread-3125] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:46,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:46,904 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:46,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:46,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:46,906 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:46,906 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:46,906 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:46,906 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:46,907 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:46,907 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:46,908 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:46,908 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:46,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:46,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:46,909 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:46,909 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:46,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:46,909 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:46,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:46,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:46,910 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:46,910 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:46,910 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:46,910 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:46,910 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:46,910 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:46,910 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:46,910 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:46,910 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:46,910 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. started... 2023-04-12 15:55:46,910 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. as already flushing 2023-04-12 15:55:46,911 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:46,911 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:46,911 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:46,911 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:46,911 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:46,911 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:46,914 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:46,914 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:46,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:46,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:46,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:46,914 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:46,914 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:46,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:46,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:46,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:46,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:46,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:46,915 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:46,915 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c54c902[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:46,916 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:46,916 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:46,917 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:46,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:46,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:46,917 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:46,917 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:46,917 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:46,917 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:46,917 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:46,919 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:46,919 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:46,919 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:46,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:46,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:46,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:46,919 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:46,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:46,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:46,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:46,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:46,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:46,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:46,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:46,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:46,921 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:46,921 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:46,921 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:46,921 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:46,921 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:46,921 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:46,923 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:46,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:46,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:46,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:46,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:46,924 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:46,924 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:46,924 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:46,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:46,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:46,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:46,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:46,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:46,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:46,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:46,925 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:46,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:46,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:46,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:46,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:46,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:46,929 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:46,929 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:46,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:46,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:46,929 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:46,929 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:46,929 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:46,929 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:46,929 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:46,929 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:46,929 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:46,930 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:46,930 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:46,930 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:46,930 DEBUG [Thread-3125] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:46,930 DEBUG [Thread-3125] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:46,930 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:46,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:46,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:46,973 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=178 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/B/be7919f2e0ea4dd9b6d7cbfe705a2f1b 2023-04-12 15:55:47,030 DEBUG [Thread-3125] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:47,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:47,032 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:47,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:47,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:47,033 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:47,033 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:47,033 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:47,033 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:47,034 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:47,034 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:47,035 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:47,035 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:47,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:47,036 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,036 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:47,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:47,036 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:47,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:47,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:47,037 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:47,037 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:47,037 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:47,037 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:47,037 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:47,037 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:47,037 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:47,037 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:47,037 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:47,037 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. started... 2023-04-12 15:55:47,037 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. as already flushing 2023-04-12 15:55:47,038 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:47,038 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:47,038 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:47,038 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,038 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:47,038 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:47,039 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:47,039 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:47,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:47,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:47,040 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:47,040 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:47,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:47,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:47,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:47,041 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:47,042 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e96358f[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:47,042 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:47,042 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:47,043 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:47,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:47,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:47,043 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:47,043 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,043 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:47,043 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:47,044 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:47,046 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,046 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:47,046 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:47,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:47,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:47,046 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:47,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:47,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:47,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:47,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:47,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:47,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:47,050 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:47,050 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:47,050 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:47,050 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:47,050 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:47,050 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:47,051 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:47,051 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:47,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:47,051 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:47,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:47,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:47,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:47,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:47,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:47,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:47,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:47,052 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:47,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:47,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:47,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:47,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,053 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:47,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:47,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,060 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:47,060 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:47,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:47,060 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:47,060 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:47,060 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:47,060 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:47,060 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:47,060 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:47,060 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,060 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:47,060 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:47,060 DEBUG [Thread-3125] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:47,061 DEBUG [Thread-3125] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:47,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:47,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:47,060 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:47,161 DEBUG [Thread-3125] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:47,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:47,163 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:47,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:47,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:47,164 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:47,164 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:47,164 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:47,164 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:47,165 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:47,165 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:47,167 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:47,167 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:47,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:47,168 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,168 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:47,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:47,168 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:47,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:47,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:47,169 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:47,169 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:47,169 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:47,169 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:47,169 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:47,169 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:47,169 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:47,169 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:47,169 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:47,169 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. started... 2023-04-12 15:55:47,169 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. as already flushing 2023-04-12 15:55:47,169 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:47,170 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:47,170 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:47,170 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,170 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:47,170 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:47,171 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,171 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:47,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:47,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:47,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:47,172 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:47,172 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:47,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:47,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:47,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:47,174 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:47,174 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@377b5fc9[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:47,175 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:47,175 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:47,176 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:47,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:47,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:47,176 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:47,176 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:47,176 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:47,176 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,176 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:47,178 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:47,178 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:47,178 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,178 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:47,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:47,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:47,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:47,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:47,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:47,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:47,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:47,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:47,180 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:47,180 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:47,180 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:47,181 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:47,181 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:47,181 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:47,183 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:47,183 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:47,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:47,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:47,183 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:47,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:47,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:47,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:47,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:47,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:47,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:47,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:47,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:47,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:47,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:47,185 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:47,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:47,186 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,186 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:47,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55322 deadline: 1681315007182, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,190 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:47,190 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,190 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:47,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55340 deadline: 1681315007187, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:47,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:47,190 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:47,190 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:47,190 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:47,190 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:47,190 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:47,191 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,191 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:47,190 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:47,191 DEBUG [Thread-3125] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:47,191 DEBUG [Thread-3125] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:47,190 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:47,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:47,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:47,191 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:47,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55260 deadline: 1681315007187, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,191 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:47,193 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:47,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55306 deadline: 1681315007189, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,291 DEBUG [Thread-3125] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:47,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:47,293 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:47,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:47,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:47,294 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:47,294 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:47,294 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:47,294 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:47,295 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:47,295 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:47,296 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:47,296 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:47,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:47,296 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,296 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:47,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:47,297 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:47,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:47,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:47,297 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:47,297 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:47,297 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:47,297 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:47,297 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:47,297 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:47,298 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:47,298 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:47,298 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:47,298 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. started... 2023-04-12 15:55:47,298 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. as already flushing 2023-04-12 15:55:47,298 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:47,298 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:47,298 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:47,298 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,298 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:47,298 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:47,300 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,300 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:47,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:47,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:47,300 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:47,300 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:47,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:47,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:47,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:47,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:47,301 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:47,302 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52da5126[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:47,302 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:47,302 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:47,304 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:47,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:47,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:47,304 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:47,304 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:47,304 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:47,304 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,304 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:47,306 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:47,306 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,306 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:47,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:47,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:47,306 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:47,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:47,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:47,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:47,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:47,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:47,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:47,308 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:47,308 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:47,308 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:47,308 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:47,308 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:47,308 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:47,310 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:47,310 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:47,310 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:47,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:47,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:47,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:47,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:47,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:47,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:47,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:47,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:47,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:47,311 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:47,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:47,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:47,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,312 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:47,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:47,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,315 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:47,315 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:47,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:47,315 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:47,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:47,315 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:47,315 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:47,315 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,315 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:47,315 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:47,315 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:47,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:47,315 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:47,316 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:47,316 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:47,316 DEBUG [Thread-3125] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:47,316 DEBUG [Thread-3125] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:47,383 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=178 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/C/787c417108be47179d30b0314a119c6d 2023-04-12 15:55:47,388 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/A/1ea3d3930ba44e338b3ba505142519ed as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/1ea3d3930ba44e338b3ba505142519ed 2023-04-12 15:55:47,392 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/1ea3d3930ba44e338b3ba505142519ed, entries=150, sequenceid=178, filesize=11.8 K 2023-04-12 15:55:47,393 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/B/be7919f2e0ea4dd9b6d7cbfe705a2f1b as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/be7919f2e0ea4dd9b6d7cbfe705a2f1b 2023-04-12 15:55:47,397 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/be7919f2e0ea4dd9b6d7cbfe705a2f1b, entries=150, sequenceid=178, filesize=11.8 K 2023-04-12 15:55:47,398 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/C/787c417108be47179d30b0314a119c6d as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/787c417108be47179d30b0314a119c6d 2023-04-12 15:55:47,402 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/787c417108be47179d30b0314a119c6d, entries=150, sequenceid=178, filesize=11.8 K 2023-04-12 15:55:47,402 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 4df381dfb8e2792d3aca65faf2673530 in 861ms, sequenceid=178, compaction requested=true 2023-04-12 15:55:47,402 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4df381dfb8e2792d3aca65faf2673530: 2023-04-12 15:55:47,402 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:55:47,403 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:55:47,403 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 15:55:47,403 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 15:55:47,403 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:55:47,403 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36450 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 15:55:47,404 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36450 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 15:55:47,404 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1912): 4df381dfb8e2792d3aca65faf2673530/A is initiating minor compaction (all files) 2023-04-12 15:55:47,404 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1912): 4df381dfb8e2792d3aca65faf2673530/B is initiating minor compaction (all files) 2023-04-12 15:55:47,404 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4df381dfb8e2792d3aca65faf2673530/A in TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:47,404 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4df381dfb8e2792d3aca65faf2673530/B in TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:47,404 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/b01e45e87f434bf1bb0769639a1bbda6, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/826fceaef48449bf93a2341960586ea9, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/1ea3d3930ba44e338b3ba505142519ed] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp, totalSize=35.6 K 2023-04-12 15:55:47,404 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/1b2b9160212044558619e4e1f26dcd66, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/870e6f9636cc4d209e233b8b7e62e3ab, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/be7919f2e0ea4dd9b6d7cbfe705a2f1b] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp, totalSize=35.6 K 2023-04-12 15:55:47,404 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting b01e45e87f434bf1bb0769639a1bbda6, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=140, earliestPutTs=1681314945194 2023-04-12 15:55:47,405 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 1b2b9160212044558619e4e1f26dcd66, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=140, earliestPutTs=1681314945194 2023-04-12 15:55:47,405 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 826fceaef48449bf93a2341960586ea9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=159, earliestPutTs=1681314945582 2023-04-12 15:55:47,405 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 1ea3d3930ba44e338b3ba505142519ed, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=178, earliestPutTs=1681314946224 2023-04-12 15:55:47,405 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 870e6f9636cc4d209e233b8b7e62e3ab, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=159, earliestPutTs=1681314945582 2023-04-12 15:55:47,406 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting be7919f2e0ea4dd9b6d7cbfe705a2f1b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=178, earliestPutTs=1681314946224 2023-04-12 15:55:47,416 DEBUG [Thread-3125] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:47,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:47,418 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:47,429 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4df381dfb8e2792d3aca65faf2673530#A#compaction#437 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:55:47,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:47,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:47,434 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:47,434 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:47,434 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:47,434 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:47,435 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:47,435 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:47,437 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:47,437 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:47,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:47,437 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,437 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:47,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:47,438 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:47,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:47,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:47,438 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:47,439 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] throttle.PressureAwareThroughputController(145): 4df381dfb8e2792d3aca65faf2673530#B#compaction#438 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:55:47,444 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:47,444 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:47,444 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:47,444 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:47,444 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:47,460 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:47,460 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:47,461 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:47,461 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. started... 2023-04-12 15:55:47,461 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4df381dfb8e2792d3aca65faf2673530 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-12 15:55:47,461 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=A 2023-04-12 15:55:47,461 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:47,461 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=B 2023-04-12 15:55:47,461 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:47,461 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=C 2023-04-12 15:55:47,461 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:47,485 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=193 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/A/8f809a4c135e4c50b825f72bde5d7779 2023-04-12 15:55:47,505 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=193 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/B/6cd1c3b7e2924717bbeb45b7d634593f 2023-04-12 15:55:47,521 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=193 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/C/31a21a91e34f4f60b6d0c1b45f7f00e6 2023-04-12 15:55:47,525 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/A/8f809a4c135e4c50b825f72bde5d7779 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/8f809a4c135e4c50b825f72bde5d7779 2023-04-12 15:55:47,529 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/8f809a4c135e4c50b825f72bde5d7779, entries=150, sequenceid=193, filesize=11.8 K 2023-04-12 15:55:47,530 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/B/6cd1c3b7e2924717bbeb45b7d634593f as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/6cd1c3b7e2924717bbeb45b7d634593f 2023-04-12 15:55:47,534 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/6cd1c3b7e2924717bbeb45b7d634593f, entries=150, sequenceid=193, filesize=11.8 K 2023-04-12 15:55:47,535 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/C/31a21a91e34f4f60b6d0c1b45f7f00e6 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/31a21a91e34f4f60b6d0c1b45f7f00e6 2023-04-12 15:55:47,538 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/31a21a91e34f4f60b6d0c1b45f7f00e6, entries=150, sequenceid=193, filesize=11.8 K 2023-04-12 15:55:47,538 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=0 B/0 for 4df381dfb8e2792d3aca65faf2673530 in 77ms, sequenceid=193, compaction requested=true 2023-04-12 15:55:47,539 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4df381dfb8e2792d3aca65faf2673530: 2023-04-12 15:55:47,539 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:47,539 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:47,539 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:47,539 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,539 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:47,539 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:47,541 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:47,541 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:47,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:47,541 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:47,541 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:47,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:47,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:47,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:47,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:47,544 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:47,544 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78ec8ea6[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:47,544 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:47,544 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:47,545 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:47,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:47,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:47,545 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:47,545 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:47,545 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:47,547 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,547 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:47,547 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:47,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:47,548 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:47,548 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:47,548 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:47,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:47,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:47,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:47,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:47,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:47,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:47,552 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:47,552 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:47,552 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:47,552 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:47,552 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:47,552 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:47,556 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:47,556 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:47,556 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:47,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:47,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:47,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:47,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:47,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:47,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:47,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:47,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:47,557 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:47,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:47,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:47,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:47,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,559 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:47,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:47,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,563 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,563 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:47,563 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:47,564 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:47,564 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:47,564 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:47,564 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:47,564 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,564 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:47,564 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:47,564 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:47,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:47,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:47,564 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:47,564 DEBUG [Thread-3125] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:47,564 DEBUG [Thread-3125] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:47,564 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:47,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:47,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:47,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on 4df381dfb8e2792d3aca65faf2673530 2023-04-12 15:55:47,636 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4df381dfb8e2792d3aca65faf2673530 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-12 15:55:47,636 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=A 2023-04-12 15:55:47,636 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:47,636 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=B 2023-04-12 15:55:47,636 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:47,636 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=C 2023-04-12 15:55:47,636 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:47,642 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=203 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/A/3839c82393bb4b51a9bc8caa97c84d5f 2023-04-12 15:55:47,651 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=203 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/B/e69ee74df29a4c06abfae7e87032f901 2023-04-12 15:55:47,664 DEBUG [Thread-3125] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:47,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:47,665 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=203 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/C/35c59df13bdb406d80baefe235282e4f 2023-04-12 15:55:47,667 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:47,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:47,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:47,668 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:47,669 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:47,669 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:47,669 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:47,669 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:47,669 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:47,670 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,670 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/A/3839c82393bb4b51a9bc8caa97c84d5f as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/3839c82393bb4b51a9bc8caa97c84d5f 2023-04-12 15:55:47,670 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:47,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:47,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:47,671 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,671 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:47,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:47,672 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:47,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:47,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:47,673 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:47,673 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:47,673 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:47,673 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:47,673 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:47,673 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:47,674 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:47,674 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:47,674 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:47,674 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. started... 2023-04-12 15:55:47,674 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. as already flushing 2023-04-12 15:55:47,674 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:47,674 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:47,674 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:47,674 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,674 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:47,674 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:47,676 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/3839c82393bb4b51a9bc8caa97c84d5f, entries=200, sequenceid=203, filesize=14.1 K 2023-04-12 15:55:47,676 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/B/e69ee74df29a4c06abfae7e87032f901 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/e69ee74df29a4c06abfae7e87032f901 2023-04-12 15:55:47,677 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,677 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:47,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:47,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:47,677 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:47,677 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:47,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:47,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:47,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:47,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,680 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/e69ee74df29a4c06abfae7e87032f901, entries=150, sequenceid=203, filesize=11.8 K 2023-04-12 15:55:47,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:47,685 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/C/35c59df13bdb406d80baefe235282e4f as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/35c59df13bdb406d80baefe235282e4f 2023-04-12 15:55:47,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:47,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47e9e679[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:47,685 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:47,685 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:47,687 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:47,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:47,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:47,687 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:47,687 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:47,687 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:47,687 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,687 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:47,689 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,689 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:47,689 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:47,689 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:47,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:47,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:47,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:47,689 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/35c59df13bdb406d80baefe235282e4f, entries=150, sequenceid=203, filesize=11.8 K 2023-04-12 15:55:47,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:47,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:47,690 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=53.67 KB/54960 for 4df381dfb8e2792d3aca65faf2673530 in 54ms, sequenceid=203, compaction requested=true 2023-04-12 15:55:47,690 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4df381dfb8e2792d3aca65faf2673530: 2023-04-12 15:55:47,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,690 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-12 15:55:47,690 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-04-12 15:55:47,690 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=4), splitQueue=0 2023-04-12 15:55:47,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:47,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on 4df381dfb8e2792d3aca65faf2673530 2023-04-12 15:55:47,692 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4df381dfb8e2792d3aca65faf2673530 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-12 15:55:47,693 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=A 2023-04-12 15:55:47,693 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:47,693 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=B 2023-04-12 15:55:47,693 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:47,693 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=C 2023-04-12 15:55:47,693 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:47,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:47,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:47,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:47,694 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:47,694 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:47,694 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:47,694 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:47,694 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:47,697 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:47,697 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:47,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:47,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:47,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:47,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:47,697 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:47,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:47,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:47,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:47,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:47,697 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:47,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:47,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:47,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:47,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,699 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:47,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:47,700 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=215 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/A/4c27ba2711f94a9e9343306b2bb30083 2023-04-12 15:55:47,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,703 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:47,703 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:47,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:47,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:47,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:47,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:47,704 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,704 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:47,704 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:47,704 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,704 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:47,704 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:47,704 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:47,704 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:47,704 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:47,704 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:47,705 DEBUG [Thread-3125] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:47,705 DEBUG [Thread-3125] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:47,709 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=215 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/B/245d539093b24f10a1a2fee1ebff2d12 2023-04-12 15:55:47,722 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=215 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/C/ec2830ae4b494b0baeeccb2cdf5a48f2 2023-04-12 15:55:47,728 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/A/4c27ba2711f94a9e9343306b2bb30083 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/4c27ba2711f94a9e9343306b2bb30083 2023-04-12 15:55:47,731 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/4c27ba2711f94a9e9343306b2bb30083, entries=250, sequenceid=215, filesize=16.4 K 2023-04-12 15:55:47,732 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/B/245d539093b24f10a1a2fee1ebff2d12 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/245d539093b24f10a1a2fee1ebff2d12 2023-04-12 15:55:47,736 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/245d539093b24f10a1a2fee1ebff2d12, entries=150, sequenceid=215, filesize=11.8 K 2023-04-12 15:55:47,740 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/C/ec2830ae4b494b0baeeccb2cdf5a48f2 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/ec2830ae4b494b0baeeccb2cdf5a48f2 2023-04-12 15:55:47,745 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:47,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55306 deadline: 1681315007728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,745 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:47,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55260 deadline: 1681315007735, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,746 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/ec2830ae4b494b0baeeccb2cdf5a48f2, entries=150, sequenceid=215, filesize=11.8 K 2023-04-12 15:55:47,747 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 4df381dfb8e2792d3aca65faf2673530 in 55ms, sequenceid=215, compaction requested=true 2023-04-12 15:55:47,747 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4df381dfb8e2792d3aca65faf2673530: 2023-04-12 15:55:47,747 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=5), splitQueue=0 2023-04-12 15:55:47,747 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=6), splitQueue=0 2023-04-12 15:55:47,747 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=7), splitQueue=0 2023-04-12 15:55:47,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on 4df381dfb8e2792d3aca65faf2673530 2023-04-12 15:55:47,757 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4df381dfb8e2792d3aca65faf2673530 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-12 15:55:47,757 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681314947734, max timestamp=1681314947755Num uniques 859; 2023-04-12 15:55:47,757 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-12 15:55:47,757 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=A 2023-04-12 15:55:47,757 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681314947734, max timestamp=1681314947755Num uniques 859; 2023-04-12 15:55:47,757 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-12 15:55:47,757 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:55:47,757 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=B 2023-04-12 15:55:47,758 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:55:47,758 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=C 2023-04-12 15:55:47,758 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681314947734, max timestamp=1681314947755Num uniques 859; 2023-04-12 15:55:47,758 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-12 15:55:47,758 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:55:47,775 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=237 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/A/7f74df07ee634e07a9d203b9293c351a 2023-04-12 15:55:47,805 DEBUG [Thread-3125] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:47,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:47,807 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:47,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:47,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:47,808 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:47,808 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:47,808 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:47,808 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:47,808 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:47,808 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:47,810 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:47,810 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:47,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:47,810 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,810 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:47,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:47,811 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:47,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:47,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:47,811 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:47,811 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:47,811 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:47,811 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:47,811 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:47,811 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:47,812 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:47,812 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:47,812 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:47,812 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. started... 2023-04-12 15:55:47,812 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. as already flushing 2023-04-12 15:55:47,812 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:47,812 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:47,812 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:47,812 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,812 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:47,812 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:47,814 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,814 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:47,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:47,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:47,815 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:47,815 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:47,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:47,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:47,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:47,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:47,816 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:47,816 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a016c5a[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:47,816 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:47,816 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:47,818 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681314947788, max timestamp=1681314947816Num uniques 859; 2023-04-12 15:55:47,818 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-12 15:55:47,818 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681314947788, max timestamp=1681314947816Num uniques 859; 2023-04-12 15:55:47,818 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681314947788, max timestamp=1681314947816Num uniques 859; 2023-04-12 15:55:47,818 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-12 15:55:47,818 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:47,818 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-12 15:55:47,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:47,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:47,819 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:47,819 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:47,819 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:47,819 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,819 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:47,821 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,821 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:47,821 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:47,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:47,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:47,822 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:47,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:47,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:47,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:47,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:47,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:47,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:47,824 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:47,824 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:47,824 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:47,824 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:47,824 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:47,824 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:47,825 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:47,825 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:47,825 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:47,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:47,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:47,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:47,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:47,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:47,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:47,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:47,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:47,826 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:47,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:47,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:47,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:47,827 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:47,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:47,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,830 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,830 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:47,830 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:47,830 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:47,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:47,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:47,830 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:47,830 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:47,831 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:47,830 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:47,831 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:47,831 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,831 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:47,831 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:47,831 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:47,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:47,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:47,831 DEBUG [Thread-3125] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:47,831 DEBUG [Thread-3125] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:47,860 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/A/980d6114b8fe45428d3c143704798f6a as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/980d6114b8fe45428d3c143704798f6a 2023-04-12 15:55:47,863 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4df381dfb8e2792d3aca65faf2673530/A of 4df381dfb8e2792d3aca65faf2673530 into 980d6114b8fe45428d3c143704798f6a(size=12.2 K), total size for store is 54.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:55:47,863 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4df381dfb8e2792d3aca65faf2673530: 2023-04-12 15:55:47,863 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530., storeName=4df381dfb8e2792d3aca65faf2673530/A, priority=13, startTime=1681314947402; duration=0sec 2023-04-12 15:55:47,864 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=7), splitQueue=0 2023-04-12 15:55:47,864 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-12 15:55:47,865 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 72558 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-12 15:55:47,865 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1912): 4df381dfb8e2792d3aca65faf2673530/C is initiating minor compaction (all files) 2023-04-12 15:55:47,865 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4df381dfb8e2792d3aca65faf2673530/C in TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:47,865 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/5b7ec464d1624b718927e56e7170276e, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/415c0b98489944ebb2e83e4e1e67bc33, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/787c417108be47179d30b0314a119c6d, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/31a21a91e34f4f60b6d0c1b45f7f00e6, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/35c59df13bdb406d80baefe235282e4f, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/ec2830ae4b494b0baeeccb2cdf5a48f2] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp, totalSize=70.9 K 2023-04-12 15:55:47,865 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 5b7ec464d1624b718927e56e7170276e, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=140, earliestPutTs=1681314945194 2023-04-12 15:55:47,866 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 415c0b98489944ebb2e83e4e1e67bc33, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=159, earliestPutTs=1681314945582 2023-04-12 15:55:47,866 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 787c417108be47179d30b0314a119c6d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=178, earliestPutTs=1681314946224 2023-04-12 15:55:47,866 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 31a21a91e34f4f60b6d0c1b45f7f00e6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=193, earliestPutTs=1681314946557 2023-04-12 15:55:47,866 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 35c59df13bdb406d80baefe235282e4f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1681314947617 2023-04-12 15:55:47,867 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting ec2830ae4b494b0baeeccb2cdf5a48f2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=215, earliestPutTs=1681314947662 2023-04-12 15:55:47,875 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4df381dfb8e2792d3aca65faf2673530#C#compaction#450 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:55:47,877 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681314947856, max timestamp=1681314947875Num uniques 1018; 2023-04-12 15:55:47,877 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681314947856, max timestamp=1681314947875Num uniques 1018; 2023-04-12 15:55:47,877 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681314947856, max timestamp=1681314947875Num uniques 1018; 2023-04-12 15:55:47,878 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-12 15:55:47,877 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-12 15:55:47,878 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-12 15:55:47,879 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/B/75b5ec3a66644cdebbcc4ed8f0c811c9 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/75b5ec3a66644cdebbcc4ed8f0c811c9 2023-04-12 15:55:47,885 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4df381dfb8e2792d3aca65faf2673530/B of 4df381dfb8e2792d3aca65faf2673530 into 75b5ec3a66644cdebbcc4ed8f0c811c9(size=12.2 K), total size for store is 47.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:55:47,885 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4df381dfb8e2792d3aca65faf2673530: 2023-04-12 15:55:47,885 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530., storeName=4df381dfb8e2792d3aca65faf2673530/B, priority=13, startTime=1681314947402; duration=0sec 2023-04-12 15:55:47,885 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=6), splitQueue=0 2023-04-12 15:55:47,886 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 6 compacting, 0 eligible, 16 blocking 2023-04-12 15:55:47,886 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-12 15:55:47,886 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-12 15:55:47,886 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. because compaction request was cancelled 2023-04-12 15:55:47,886 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-12 15:55:47,888 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/C/58d42b1c4659480ba53d994a27c6290e as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/58d42b1c4659480ba53d994a27c6290e 2023-04-12 15:55:47,888 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 55758 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-12 15:55:47,888 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1912): 4df381dfb8e2792d3aca65faf2673530/A is initiating minor compaction (all files) 2023-04-12 15:55:47,889 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4df381dfb8e2792d3aca65faf2673530/A in TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:47,889 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/980d6114b8fe45428d3c143704798f6a, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/8f809a4c135e4c50b825f72bde5d7779, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/3839c82393bb4b51a9bc8caa97c84d5f, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/4c27ba2711f94a9e9343306b2bb30083] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp, totalSize=54.5 K 2023-04-12 15:55:47,889 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 980d6114b8fe45428d3c143704798f6a, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=178, earliestPutTs=1681314946224 2023-04-12 15:55:47,889 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 8f809a4c135e4c50b825f72bde5d7779, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=193, earliestPutTs=1681314946557 2023-04-12 15:55:47,890 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 3839c82393bb4b51a9bc8caa97c84d5f, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1681314947617 2023-04-12 15:55:47,890 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 4c27ba2711f94a9e9343306b2bb30083, keycount=250, bloomtype=ROW, size=16.4 K, encoding=NONE, compression=NONE, seqNum=215, earliestPutTs=1681314947662 2023-04-12 15:55:47,893 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 4df381dfb8e2792d3aca65faf2673530/C of 4df381dfb8e2792d3aca65faf2673530 into 58d42b1c4659480ba53d994a27c6290e(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:55:47,893 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4df381dfb8e2792d3aca65faf2673530: 2023-04-12 15:55:47,893 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530., storeName=4df381dfb8e2792d3aca65faf2673530/C, priority=10, startTime=1681314947747; duration=0sec 2023-04-12 15:55:47,893 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=4), splitQueue=0 2023-04-12 15:55:47,893 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-12 15:55:47,895 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48588 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-12 15:55:47,895 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1912): 4df381dfb8e2792d3aca65faf2673530/B is initiating minor compaction (all files) 2023-04-12 15:55:47,895 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4df381dfb8e2792d3aca65faf2673530/B in TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:47,895 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/75b5ec3a66644cdebbcc4ed8f0c811c9, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/6cd1c3b7e2924717bbeb45b7d634593f, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/e69ee74df29a4c06abfae7e87032f901, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/245d539093b24f10a1a2fee1ebff2d12] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp, totalSize=47.4 K 2023-04-12 15:55:47,896 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 75b5ec3a66644cdebbcc4ed8f0c811c9, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=178, earliestPutTs=1681314946224 2023-04-12 15:55:47,897 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 6cd1c3b7e2924717bbeb45b7d634593f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=193, earliestPutTs=1681314946557 2023-04-12 15:55:47,897 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting e69ee74df29a4c06abfae7e87032f901, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1681314947617 2023-04-12 15:55:47,897 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 245d539093b24f10a1a2fee1ebff2d12, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=215, earliestPutTs=1681314947662 2023-04-12 15:55:47,902 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] throttle.PressureAwareThroughputController(145): 4df381dfb8e2792d3aca65faf2673530#A#compaction#451 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:55:47,910 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4df381dfb8e2792d3aca65faf2673530#B#compaction#452 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:55:47,916 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/B/2ba63acda7614592825627ab3aa42793 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/2ba63acda7614592825627ab3aa42793 2023-04-12 15:55:47,921 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:47,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55340 deadline: 1681315007908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,921 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:47,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55322 deadline: 1681315007909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,922 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=327, cellSize=14962, totalHeapSize=34248, min timestamp=1681314947901, max timestamp=1681314947919Num uniques 1077; 2023-04-12 15:55:47,922 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=327, cellSize=14972, totalHeapSize=34248, min timestamp=1681314947901, max timestamp=1681314947919Num uniques 1077; 2023-04-12 15:55:47,922 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=327, cellSize=14962, totalHeapSize=34248, min timestamp=1681314947901, max timestamp=1681314947919Num uniques 1077; 2023-04-12 15:55:47,922 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=14972, suffix heap size=139976,new segment heap size=34248 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=327 2023-04-12 15:55:47,922 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49312, new segment data size=14962, suffix heap size=139976,new segment heap size=34248 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=327 2023-04-12 15:55:47,922 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49312, new segment data size=14962, suffix heap size=139976,new segment heap size=34248 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=327 2023-04-12 15:55:47,924 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 4df381dfb8e2792d3aca65faf2673530/B of 4df381dfb8e2792d3aca65faf2673530 into 2ba63acda7614592825627ab3aa42793(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:55:47,924 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4df381dfb8e2792d3aca65faf2673530: 2023-04-12 15:55:47,924 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530., storeName=4df381dfb8e2792d3aca65faf2673530/B, priority=12, startTime=1681314947690; duration=0sec 2023-04-12 15:55:47,924 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-04-12 15:55:47,924 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 4 compacting, 0 eligible, 16 blocking 2023-04-12 15:55:47,924 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-12 15:55:47,924 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-12 15:55:47,925 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. because compaction request was cancelled 2023-04-12 15:55:47,925 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 1 store files, 0 compacting, 1 eligible, 16 blocking 2023-04-12 15:55:47,925 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-12 15:55:47,925 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-12 15:55:47,925 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. because compaction request was cancelled 2023-04-12 15:55:47,925 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 1 store files, 0 compacting, 1 eligible, 16 blocking 2023-04-12 15:55:47,926 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-12 15:55:47,926 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-12 15:55:47,926 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. because compaction request was cancelled 2023-04-12 15:55:47,931 DEBUG [Thread-3125] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:47,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:47,933 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:47,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:47,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:47,934 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:47,934 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:47,934 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:47,934 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:47,935 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:47,935 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:47,936 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:47,936 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:47,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:47,936 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,936 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:47,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:47,937 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:47,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:47,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:47,937 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:47,937 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:47,937 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:47,937 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:47,937 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:47,938 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:47,938 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:47,938 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:47,938 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:47,938 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. started... 2023-04-12 15:55:47,938 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. as already flushing 2023-04-12 15:55:47,938 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:47,938 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:47,938 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:47,938 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,938 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:47,938 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:47,941 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,941 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:47,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:47,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:47,941 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:47,942 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:47,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:47,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:47,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:47,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:47,943 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:47,943 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:47,943 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a524b5d[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:47,943 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:47,944 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:47,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:47,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:47,944 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:47,944 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:47,944 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,944 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:47,944 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:47,945 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,945 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:47,946 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:47,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:47,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:47,946 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:47,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:47,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:47,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:47,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:47,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:47,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:47,948 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:47,948 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:47,948 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:47,948 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:47,948 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:47,948 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:47,949 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:47,949 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:47,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:47,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:47,949 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:47,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:47,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:47,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:47,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:47,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:47,950 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:47,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:47,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:47,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:47,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:47,951 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:47,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:47,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,954 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:47,954 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,954 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:47,954 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:47,955 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:47,954 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:47,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:47,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:47,954 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:47,955 DEBUG [Thread-3125] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:47,955 DEBUG [Thread-3125] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:47,955 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:47,955 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:47,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:47,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:47,955 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:47,955 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:47,955 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:47,955 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:48,001 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:48,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55260 deadline: 1681315007993, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,007 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:48,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55306 deadline: 1681315008000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,008 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:48,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55336 deadline: 1681315008001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,025 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:48,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55340 deadline: 1681315008022, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,025 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:48,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55322 deadline: 1681315008023, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,055 DEBUG [Thread-3125] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:48,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:48,057 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:48,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:48,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:48,059 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:48,059 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:48,059 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:48,059 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:48,059 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:48,059 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:48,061 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:48,061 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:48,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:48,061 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,061 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:48,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:48,062 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:48,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:48,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:48,062 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:48,062 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:48,062 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:48,062 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:48,062 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:48,062 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:48,063 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:48,063 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:48,063 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:48,063 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. started... 2023-04-12 15:55:48,063 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. as already flushing 2023-04-12 15:55:48,063 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:48,063 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:48,063 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:48,063 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,063 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:48,063 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:48,067 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,067 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:48,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:48,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:48,068 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:48,068 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:48,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:48,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:48,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:48,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:48,069 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:48,069 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:48,069 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:48,069 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3657790f[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:48,074 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:48,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:48,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:48,074 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:48,074 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:48,074 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:48,074 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,074 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:48,075 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:48,075 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:48,075 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:48,075 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:48,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:48,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:48,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:48,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:48,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:48,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:48,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:48,078 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:48,078 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:48,078 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:48,078 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:48,078 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:48,078 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:48,080 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:48,080 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:48,080 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:48,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:48,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:48,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:48,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:48,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:48,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:48,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:48,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:48,082 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:48,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:48,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:48,083 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:48,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:48,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:48,085 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:48,085 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:48,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:48,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:48,085 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,086 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:48,086 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:48,086 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,086 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:48,086 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:48,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:48,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:48,086 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:48,086 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:48,086 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:48,086 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:48,086 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:48,086 DEBUG [Thread-3125] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:48,086 DEBUG [Thread-3125] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:48,107 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:48,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55260 deadline: 1681315008102, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,115 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:48,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55336 deadline: 1681315008109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,115 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:48,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55306 deadline: 1681315008109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,183 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=237 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/B/6076d11358024647bddf2f498c26e762 2023-04-12 15:55:48,186 DEBUG [Thread-3125] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:48,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:48,189 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:48,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:48,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:48,190 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:48,190 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:48,190 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:48,190 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:48,191 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:48,191 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:48,194 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:48,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:48,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:48,194 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:48,197 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,197 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:48,198 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:48,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:48,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:48,198 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:48,198 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:48,198 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:48,198 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:48,198 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:48,198 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:48,199 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:48,199 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:48,199 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:48,199 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. started... 2023-04-12 15:55:48,199 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. as already flushing 2023-04-12 15:55:48,199 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:48,199 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:48,199 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:48,199 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,199 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:48,199 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:48,201 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:48,201 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:48,201 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:48,203 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:48,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:48,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:48,204 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=237 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/C/cc899cecff9c4628bf33e7a71989a13c 2023-04-12 15:55:48,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:48,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:48,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:48,205 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:48,205 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e058b92[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:48,205 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:48,205 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:48,207 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:48,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:48,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:48,207 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,207 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:48,208 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:48,208 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:48,208 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:48,210 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:48,210 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:48,210 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,210 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:48,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:48,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:48,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:48,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:48,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:48,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:48,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:48,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:48,213 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:48,213 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:48,213 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:48,213 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:48,213 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:48,213 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:48,215 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:48,215 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:48,215 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:48,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:48,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:48,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:48,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:48,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:48,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:48,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:48,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:48,216 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:48,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:48,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:48,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:48,218 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:48,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:48,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,223 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:48,223 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:48,223 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,223 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:48,223 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:48,223 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,223 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:48,223 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:48,223 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:48,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:48,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:48,224 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:48,224 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:48,224 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:48,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:48,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:48,224 DEBUG [Thread-3125] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:48,224 DEBUG [Thread-3125] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:48,229 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:48,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55340 deadline: 1681315008226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,229 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:48,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55322 deadline: 1681315008227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,231 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/A/7f74df07ee634e07a9d203b9293c351a as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/7f74df07ee634e07a9d203b9293c351a 2023-04-12 15:55:48,237 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/7f74df07ee634e07a9d203b9293c351a, entries=250, sequenceid=237, filesize=16.4 K 2023-04-12 15:55:48,237 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/B/6076d11358024647bddf2f498c26e762 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/6076d11358024647bddf2f498c26e762 2023-04-12 15:55:48,246 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/6076d11358024647bddf2f498c26e762, entries=150, sequenceid=237, filesize=11.8 K 2023-04-12 15:55:48,247 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/C/cc899cecff9c4628bf33e7a71989a13c as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/cc899cecff9c4628bf33e7a71989a13c 2023-04-12 15:55:48,267 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/cc899cecff9c4628bf33e7a71989a13c, entries=150, sequenceid=237, filesize=11.8 K 2023-04-12 15:55:48,268 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~86.11 KB/88176, currentSize=154.31 KB/158010 for 4df381dfb8e2792d3aca65faf2673530 in 511ms, sequenceid=237, compaction requested=false 2023-04-12 15:55:48,268 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4df381dfb8e2792d3aca65faf2673530: 2023-04-12 15:55:48,313 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/A/44d2d6670b8e4b7aaa2a23ac39753db2 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/44d2d6670b8e4b7aaa2a23ac39753db2 2023-04-12 15:55:48,318 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681314947953, max timestamp=1681314948315Num uniques 1186; 2023-04-12 15:55:48,318 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=54316, new segment data size=8516, suffix heap size=151312,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1186, new segment cells count=186 2023-04-12 15:55:48,320 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 4df381dfb8e2792d3aca65faf2673530/A of 4df381dfb8e2792d3aca65faf2673530 into 44d2d6670b8e4b7aaa2a23ac39753db2(size=12.3 K), total size for store is 28.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:55:48,320 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4df381dfb8e2792d3aca65faf2673530: 2023-04-12 15:55:48,320 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530., storeName=4df381dfb8e2792d3aca65faf2673530/A, priority=12, startTime=1681314947747; duration=0sec 2023-04-12 15:55:48,320 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:55:48,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on 4df381dfb8e2792d3aca65faf2673530 2023-04-12 15:55:48,320 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4df381dfb8e2792d3aca65faf2673530 3/3 column families, dataSize=116.29 KB heapSize=320.98 KB 2023-04-12 15:55:48,320 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=A 2023-04-12 15:55:48,321 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:55:48,321 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=B 2023-04-12 15:55:48,321 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-12 15:55:48,321 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=C 2023-04-12 15:55:48,321 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-12 15:55:48,325 DEBUG [Thread-3125] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:48,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:48,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:48,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:48,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:48,329 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:48,329 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:48,329 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:48,329 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:48,330 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:48,330 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:48,332 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:48,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:48,332 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:48,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:48,333 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,333 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:48,333 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:48,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:48,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:48,334 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:48,348 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:48,348 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:48,348 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:48,348 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:48,348 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:48,356 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:48,356 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:48,356 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:48,356 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. started... 2023-04-12 15:55:48,356 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. as already flushing 2023-04-12 15:55:48,356 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:48,356 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:48,356 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:48,356 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,356 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:48,356 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:48,359 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:48,362 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:48,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:48,363 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:48,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:48,363 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=314 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/A/db153c3d9c604c89929693d2875d565d 2023-04-12 15:55:48,363 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:48,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:48,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:48,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:48,364 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:48,365 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1fc743b4[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:48,365 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:48,365 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:48,366 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:48,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:48,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:48,366 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:48,366 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:48,366 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:48,366 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,366 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:48,368 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:48,368 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:48,368 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:48,369 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:48,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:48,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:48,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:48,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:48,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:48,373 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=314 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/B/9179ee7e9ad24dafbfa8a9336dada40c 2023-04-12 15:55:48,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:48,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:48,374 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:48,374 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:48,374 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:48,374 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:48,374 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:48,374 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:48,376 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:48,376 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:48,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:48,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:48,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:48,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:48,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:48,376 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:48,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:48,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:48,377 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:48,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:48,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:48,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:48,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:48,378 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:48,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:48,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,382 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,382 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:48,382 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:48,382 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:48,382 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:48,382 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:48,382 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:48,382 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,382 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:48,382 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:48,382 DEBUG [Thread-3125] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:48,382 DEBUG [Thread-3125] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:48,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:48,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:48,382 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:48,382 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:48,382 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:48,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:48,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:48,397 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:48,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55306 deadline: 1681315008386, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,397 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:48,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55260 deadline: 1681315008386, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,398 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:48,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55336 deadline: 1681315008386, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,482 DEBUG [Thread-3125] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:48,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:48,484 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:48,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:48,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:48,486 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:48,486 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:48,486 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:48,486 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:48,486 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:48,486 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:48,488 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,488 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:48,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:48,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:48,488 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,488 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:48,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:48,488 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:48,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:48,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:48,489 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:48,489 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:48,489 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:48,489 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:48,489 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:48,489 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:48,489 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:48,489 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:48,489 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:48,489 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. started... 2023-04-12 15:55:48,490 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. as already flushing 2023-04-12 15:55:48,490 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:48,490 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:48,490 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:48,490 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,490 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:48,490 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:48,491 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:48,491 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:48,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:48,492 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:48,492 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:48,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:48,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:48,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:48,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:48,493 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:48,493 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27f7ebb2[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:48,493 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:48,493 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:48,495 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:48,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:48,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:48,495 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:48,495 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,495 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:48,495 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:48,495 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:48,497 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:48,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:48,497 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:48,497 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:48,497 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:48,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:48,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:48,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:48,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:48,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:48,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:48,500 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:48,500 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:48,500 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:48,500 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:48,500 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:48,500 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:48,502 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:48,502 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:48,502 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:48,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:48,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:48,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:48,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:48,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:48,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:48,502 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:48,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:48,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:48,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:48,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:48,504 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:48,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:48,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:48,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,506 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,506 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:48,506 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:48,506 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:48,506 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:48,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:48,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:48,506 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,506 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:48,506 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:48,506 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:48,506 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:48,506 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:48,506 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:48,507 DEBUG [Thread-3125] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:48,507 DEBUG [Thread-3125] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:48,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:48,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:48,507 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:48,507 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:48,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55306 deadline: 1681315008499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,508 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:48,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55260 deadline: 1681315008499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,508 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:48,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55336 deadline: 1681315008499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,532 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:48,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55340 deadline: 1681315008531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,532 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:48,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55322 deadline: 1681315008531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,607 DEBUG [Thread-3125] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:48,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:48,609 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:48,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:48,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:48,610 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:48,610 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:48,610 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:48,610 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:48,611 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:48,611 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:48,613 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:48,613 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:48,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:48,613 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,613 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:48,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:48,614 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:48,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:48,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:48,614 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:48,614 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:48,614 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:48,614 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:48,614 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:48,615 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:48,615 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:48,615 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:48,615 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:48,615 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. started... 2023-04-12 15:55:48,615 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. as already flushing 2023-04-12 15:55:48,615 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:48,615 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:48,615 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:48,615 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,615 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:48,615 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:48,617 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:48,617 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:48,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:48,617 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:48,617 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:48,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:48,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:48,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:48,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:48,619 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:48,619 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ae57142[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:48,619 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:48,619 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:48,620 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:48,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:48,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:48,621 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:48,621 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:48,621 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:48,621 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,621 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:48,622 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,622 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:48,622 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:48,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:48,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:48,622 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:48,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:48,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:48,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:48,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:48,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:48,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:48,624 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:48,624 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:48,624 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:48,624 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:48,624 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:48,625 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:48,627 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:48,628 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:48,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:48,628 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:48,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:48,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:48,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:48,628 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:48,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:48,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:48,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:48,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:48,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:48,629 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:48,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:48,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:48,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:48,631 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,632 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:48,632 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:48,632 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:48,632 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:48,632 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:48,632 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,632 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:48,632 DEBUG [Thread-3125] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:48,632 DEBUG [Thread-3125] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:48,632 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:48,632 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:48,632 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:48,633 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:48,633 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:48,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:48,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:48,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:48,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:48,713 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:48,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55306 deadline: 1681315008709, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,714 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:48,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55336 deadline: 1681315008710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,714 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:48,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55260 deadline: 1681315008711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,732 DEBUG [Thread-3125] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:48,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:48,734 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:48,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:48,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:48,736 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:48,736 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:48,736 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:48,736 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:48,736 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:48,736 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:48,738 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:48,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:48,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:48,738 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:48,738 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,738 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:48,738 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:48,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:48,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:48,739 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:48,739 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:48,739 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:48,739 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:48,739 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:48,739 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:48,740 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:48,740 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:48,740 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:48,740 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. started... 2023-04-12 15:55:48,740 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. as already flushing 2023-04-12 15:55:48,740 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:48,740 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:48,740 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:48,740 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,740 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:48,740 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:48,743 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:48,743 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:48,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:48,743 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:48,743 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:48,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:48,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:48,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:48,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:48,745 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:48,745 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@382e2697[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:48,745 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:48,745 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:48,747 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:48,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:48,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:48,747 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:48,747 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:48,747 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:48,747 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,747 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:48,749 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,749 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:48,749 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:48,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:48,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:48,749 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:48,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:48,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:48,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:48,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:48,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:48,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:48,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:48,752 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:48,752 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:48,752 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:48,752 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:48,752 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:48,753 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:48,753 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:48,753 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:48,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:48,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:48,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:48,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:48,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:48,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:48,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:48,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:48,754 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:48,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:48,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:48,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:48,756 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:48,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:48,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,761 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:48,761 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:48,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:48,761 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:48,761 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:48,761 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:48,762 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:48,762 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:48,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:48,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:48,761 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:48,762 DEBUG [Thread-3125] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:48,762 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,762 DEBUG [Thread-3125] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:48,762 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:48,762 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:48,762 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:48,783 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=314 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/C/7a0a6ac02e964c49aea85aed5775da4d 2023-04-12 15:55:48,787 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/A/db153c3d9c604c89929693d2875d565d as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/db153c3d9c604c89929693d2875d565d 2023-04-12 15:55:48,791 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/db153c3d9c604c89929693d2875d565d, entries=200, sequenceid=314, filesize=14.3 K 2023-04-12 15:55:48,791 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/B/9179ee7e9ad24dafbfa8a9336dada40c as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/9179ee7e9ad24dafbfa8a9336dada40c 2023-04-12 15:55:48,796 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/9179ee7e9ad24dafbfa8a9336dada40c, entries=150, sequenceid=314, filesize=11.9 K 2023-04-12 15:55:48,800 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/C/7a0a6ac02e964c49aea85aed5775da4d as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/7a0a6ac02e964c49aea85aed5775da4d 2023-04-12 15:55:48,805 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/7a0a6ac02e964c49aea85aed5775da4d, entries=150, sequenceid=314, filesize=11.9 K 2023-04-12 15:55:48,806 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~116.29 KB/119080, heapSize ~320.94 KB/328640, currentSize=67.09 KB/68700 for 4df381dfb8e2792d3aca65faf2673530 in 486ms, sequenceid=314, compaction requested=true 2023-04-12 15:55:48,806 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4df381dfb8e2792d3aca65faf2673530: 2023-04-12 15:55:48,806 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:55:48,806 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 15:55:48,806 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:55:48,806 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 15:55:48,806 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:55:48,810 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 44058 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 15:55:48,810 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1912): 4df381dfb8e2792d3aca65faf2673530/A is initiating minor compaction (all files) 2023-04-12 15:55:48,810 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4df381dfb8e2792d3aca65faf2673530/A in TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:48,810 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/44d2d6670b8e4b7aaa2a23ac39753db2, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/7f74df07ee634e07a9d203b9293c351a, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/db153c3d9c604c89929693d2875d565d] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp, totalSize=43.0 K 2023-04-12 15:55:48,810 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36838 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 15:55:48,810 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1912): 4df381dfb8e2792d3aca65faf2673530/B is initiating minor compaction (all files) 2023-04-12 15:55:48,810 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4df381dfb8e2792d3aca65faf2673530/B in TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:48,810 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/2ba63acda7614592825627ab3aa42793, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/6076d11358024647bddf2f498c26e762, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/9179ee7e9ad24dafbfa8a9336dada40c] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp, totalSize=36.0 K 2023-04-12 15:55:48,811 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 44d2d6670b8e4b7aaa2a23ac39753db2, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=215, earliestPutTs=1681314947662 2023-04-12 15:55:48,811 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 2ba63acda7614592825627ab3aa42793, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=215, earliestPutTs=1681314947662 2023-04-12 15:55:48,811 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 7f74df07ee634e07a9d203b9293c351a, keycount=250, bloomtype=ROW, size=16.4 K, encoding=NONE, compression=NONE, seqNum=237, earliestPutTs=1681314947734 2023-04-12 15:55:48,811 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 6076d11358024647bddf2f498c26e762, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=237, earliestPutTs=1681314947734 2023-04-12 15:55:48,811 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting db153c3d9c604c89929693d2875d565d, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=314, earliestPutTs=1681314947953 2023-04-12 15:55:48,812 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 9179ee7e9ad24dafbfa8a9336dada40c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=314, earliestPutTs=1681314947953 2023-04-12 15:55:48,818 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] throttle.PressureAwareThroughputController(145): 4df381dfb8e2792d3aca65faf2673530#B#compaction#457 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:55:48,819 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4df381dfb8e2792d3aca65faf2673530#A#compaction#458 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:55:48,841 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/A/200eb3c214764a0bbe41dbd340b59ea4 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/200eb3c214764a0bbe41dbd340b59ea4 2023-04-12 15:55:48,845 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4df381dfb8e2792d3aca65faf2673530/A of 4df381dfb8e2792d3aca65faf2673530 into 200eb3c214764a0bbe41dbd340b59ea4(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:55:48,845 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4df381dfb8e2792d3aca65faf2673530: 2023-04-12 15:55:48,845 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530., storeName=4df381dfb8e2792d3aca65faf2673530/A, priority=13, startTime=1681314948806; duration=0sec 2023-04-12 15:55:48,846 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:55:48,846 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 15:55:48,846 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36804 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 15:55:48,847 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1912): 4df381dfb8e2792d3aca65faf2673530/C is initiating minor compaction (all files) 2023-04-12 15:55:48,847 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4df381dfb8e2792d3aca65faf2673530/C in TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:48,847 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/58d42b1c4659480ba53d994a27c6290e, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/cc899cecff9c4628bf33e7a71989a13c, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/7a0a6ac02e964c49aea85aed5775da4d] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp, totalSize=35.9 K 2023-04-12 15:55:48,847 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 58d42b1c4659480ba53d994a27c6290e, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=215, earliestPutTs=1681314947662 2023-04-12 15:55:48,848 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting cc899cecff9c4628bf33e7a71989a13c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=237, earliestPutTs=1681314947734 2023-04-12 15:55:48,848 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 7a0a6ac02e964c49aea85aed5775da4d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=314, earliestPutTs=1681314947953 2023-04-12 15:55:48,855 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4df381dfb8e2792d3aca65faf2673530#C#compaction#459 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:55:48,862 DEBUG [Thread-3125] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:48,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:48,864 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:48,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:48,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:48,865 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:48,866 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:48,866 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:48,866 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:48,866 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:48,866 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:48,867 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/C/5bffd9da81d64cf88e694fdcca264c95 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/5bffd9da81d64cf88e694fdcca264c95 2023-04-12 15:55:48,869 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,869 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:48,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:48,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:48,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:48,870 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,870 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:48,870 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:48,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:48,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:48,870 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:48,872 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:48,872 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:48,872 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:48,872 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:48,872 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:48,872 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:48,872 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:48,872 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4df381dfb8e2792d3aca65faf2673530/C of 4df381dfb8e2792d3aca65faf2673530 into 5bffd9da81d64cf88e694fdcca264c95(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:55:48,872 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4df381dfb8e2792d3aca65faf2673530: 2023-04-12 15:55:48,872 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:48,872 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. started... 2023-04-12 15:55:48,872 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530., storeName=4df381dfb8e2792d3aca65faf2673530/C, priority=13, startTime=1681314948806; duration=0sec 2023-04-12 15:55:48,873 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4df381dfb8e2792d3aca65faf2673530 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-12 15:55:48,873 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:55:48,873 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=A 2023-04-12 15:55:48,873 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:48,873 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=B 2023-04-12 15:55:48,873 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:48,873 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=C 2023-04-12 15:55:48,873 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:48,885 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=329 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/A/8da6814cb74e45b19fc42d659c2b1ff0 2023-04-12 15:55:48,899 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=329 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/B/fae4637c71464ba093f316363f9bf3f1 2023-04-12 15:55:48,913 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=329 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/C/18a9bb7e91414ccca206b47d2e80cd78 2023-04-12 15:55:48,918 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/A/8da6814cb74e45b19fc42d659c2b1ff0 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/8da6814cb74e45b19fc42d659c2b1ff0 2023-04-12 15:55:48,921 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/8da6814cb74e45b19fc42d659c2b1ff0, entries=150, sequenceid=329, filesize=11.9 K 2023-04-12 15:55:48,922 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/B/fae4637c71464ba093f316363f9bf3f1 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/fae4637c71464ba093f316363f9bf3f1 2023-04-12 15:55:48,925 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/fae4637c71464ba093f316363f9bf3f1, entries=150, sequenceid=329, filesize=11.9 K 2023-04-12 15:55:48,925 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.StoreScanner(961): StoreScanner already closing. There is no need to updateReaders 2023-04-12 15:55:48,927 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/C/18a9bb7e91414ccca206b47d2e80cd78 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/18a9bb7e91414ccca206b47d2e80cd78 2023-04-12 15:55:48,932 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/18a9bb7e91414ccca206b47d2e80cd78, entries=150, sequenceid=329, filesize=11.9 K 2023-04-12 15:55:48,932 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=0 B/0 for 4df381dfb8e2792d3aca65faf2673530 in 59ms, sequenceid=329, compaction requested=false 2023-04-12 15:55:48,932 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4df381dfb8e2792d3aca65faf2673530: 2023-04-12 15:55:48,932 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:48,932 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:48,932 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:48,932 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,933 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:48,933 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:48,934 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:48,934 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:48,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:48,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:48,935 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:48,935 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:48,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:48,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:48,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:48,938 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:48,938 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1dd118c5[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:48,938 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:48,938 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:48,940 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:48,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:48,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:48,940 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:48,940 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:48,940 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:48,942 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,942 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:48,942 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:48,942 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:48,942 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:48,943 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:48,943 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:48,943 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:48,943 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:48,943 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:48,943 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:48,948 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:48,948 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:48,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:48,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:48,948 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:48,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:48,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:48,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:48,948 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:48,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:48,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:48,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:48,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:48,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:48,951 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:48,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:48,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:48,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,953 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:48,953 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,953 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:48,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:48,953 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:48,954 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:48,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:48,953 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:48,954 DEBUG [Thread-3125] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:48,953 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:48,954 DEBUG [Thread-3125] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:48,954 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:48,954 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:48,954 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:48,954 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:48,954 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:48,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:48,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:48,956 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:49,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on 4df381dfb8e2792d3aca65faf2673530 2023-04-12 15:55:49,025 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4df381dfb8e2792d3aca65faf2673530 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-12 15:55:49,025 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=A 2023-04-12 15:55:49,025 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:49,025 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=B 2023-04-12 15:55:49,025 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:49,025 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=C 2023-04-12 15:55:49,025 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:49,033 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=339 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/A/efff4227ba284e4092a5e0a6980c92ee 2023-04-12 15:55:49,043 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=339 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/B/f80962a0689242fba1a75853d3a7ddbc 2023-04-12 15:55:49,053 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=339 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/C/a685dbb2493249d997efedbed80d4b7e 2023-04-12 15:55:49,054 DEBUG [Thread-3125] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:49,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:49,055 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681314949042, max timestamp=1681314949053Num uniques 859; 2023-04-12 15:55:49,055 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681314949042, max timestamp=1681314949053Num uniques 859; 2023-04-12 15:55:49,055 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-12 15:55:49,055 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-12 15:55:49,056 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:49,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:49,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:49,057 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:49,057 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:49,057 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:49,057 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:49,058 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:49,058 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:49,058 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/A/efff4227ba284e4092a5e0a6980c92ee as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/efff4227ba284e4092a5e0a6980c92ee 2023-04-12 15:55:49,059 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:49,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:49,059 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:49,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:49,059 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:49,059 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:49,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:49,059 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:49,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:49,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:49,060 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:49,060 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:49,060 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:49,060 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:49,060 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:49,060 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:49,060 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:49,060 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:49,060 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:49,060 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. started... 2023-04-12 15:55:49,061 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. as already flushing 2023-04-12 15:55:49,061 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:49,061 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:49,061 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:49,061 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:49,061 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:49,061 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:49,062 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/efff4227ba284e4092a5e0a6980c92ee, entries=150, sequenceid=339, filesize=11.9 K 2023-04-12 15:55:49,062 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/B/f80962a0689242fba1a75853d3a7ddbc as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/f80962a0689242fba1a75853d3a7ddbc 2023-04-12 15:55:49,063 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:49,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:49,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:49,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:49,063 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:49,064 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:49,064 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:49,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:49,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:49,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:49,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:49,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:49,065 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:49,065 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e3cab6[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:49,065 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:49,065 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:49,066 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/f80962a0689242fba1a75853d3a7ddbc, entries=150, sequenceid=339, filesize=11.9 K 2023-04-12 15:55:49,067 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/C/a685dbb2493249d997efedbed80d4b7e as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/a685dbb2493249d997efedbed80d4b7e 2023-04-12 15:55:49,068 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:49,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:49,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:49,073 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:49,073 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:49,073 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:49,073 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:49,073 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:49,075 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/a685dbb2493249d997efedbed80d4b7e, entries=150, sequenceid=339, filesize=11.9 K 2023-04-12 15:55:49,075 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:49,075 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:49,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:49,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:49,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:49,075 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:49,075 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:49,075 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=111.82 KB/114500 for 4df381dfb8e2792d3aca65faf2673530 in 50ms, sequenceid=339, compaction requested=true 2023-04-12 15:55:49,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:49,075 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4df381dfb8e2792d3aca65faf2673530: 2023-04-12 15:55:49,075 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:55:49,075 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 15:55:49,075 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:55:49,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:49,076 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-12 15:55:49,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:49,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:49,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:49,076 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37240 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 15:55:49,078 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1912): 4df381dfb8e2792d3aca65faf2673530/A is initiating minor compaction (all files) 2023-04-12 15:55:49,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:49,079 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4df381dfb8e2792d3aca65faf2673530/A in TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:49,079 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/200eb3c214764a0bbe41dbd340b59ea4, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/8da6814cb74e45b19fc42d659c2b1ff0, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/efff4227ba284e4092a5e0a6980c92ee] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp, totalSize=36.4 K 2023-04-12 15:55:49,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:49,079 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 200eb3c214764a0bbe41dbd340b59ea4, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=314, earliestPutTs=1681314947953 2023-04-12 15:55:49,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:49,080 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:49,080 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:49,080 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:49,080 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:49,080 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:49,080 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 8da6814cb74e45b19fc42d659c2b1ff0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=329, earliestPutTs=1681314948336 2023-04-12 15:55:49,080 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:49,080 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting efff4227ba284e4092a5e0a6980c92ee, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=339, earliestPutTs=1681314949022 2023-04-12 15:55:49,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on 4df381dfb8e2792d3aca65faf2673530 2023-04-12 15:55:49,081 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4df381dfb8e2792d3aca65faf2673530 3/3 column families, dataSize=118.53 KB heapSize=334.09 KB 2023-04-12 15:55:49,081 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=A 2023-04-12 15:55:49,081 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:55:49,081 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=B 2023-04-12 15:55:49,081 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:55:49,081 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=C 2023-04-12 15:55:49,081 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:55:49,082 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:49,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:49,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:49,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:49,083 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:49,083 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:49,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:49,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:49,083 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:49,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:49,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:49,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:49,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:49,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:49,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:49,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:49,085 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:49,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:49,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:49,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:49,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:49,087 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4df381dfb8e2792d3aca65faf2673530#A#compaction#467 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:55:49,089 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:49,089 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:49,089 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:49,089 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:49,089 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:49,089 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:49,089 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:49,089 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:49,089 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:49,089 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:49,089 DEBUG [Thread-3125] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:49,089 DEBUG [Thread-3125] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:49,089 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:49,090 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:49,090 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:49,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:49,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:49,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:49,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:49,099 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:49,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55340 deadline: 1681315009091, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:49,099 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:49,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55306 deadline: 1681315009092, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:49,100 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:49,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55260 deadline: 1681315009093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:49,103 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:49,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55336 deadline: 1681315009094, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:49,103 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:49,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55322 deadline: 1681315009094, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:49,190 DEBUG [Thread-3125] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:49,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:49,192 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:49,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:49,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:49,193 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:49,193 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:49,193 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:49,193 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:49,194 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:49,194 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:49,200 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:49,200 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:49,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:49,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:49,200 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:49,200 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:49,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:49,201 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:49,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:49,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:49,201 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:49,202 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:49,202 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:49,202 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:49,202 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:49,202 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:49,202 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:49,202 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:49,202 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:49,202 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. started... 2023-04-12 15:55:49,202 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. as already flushing 2023-04-12 15:55:49,202 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:49,202 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:49,202 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:49,202 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:49,202 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:49,202 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:49,204 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:49,204 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:49,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:49,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:49,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:49,204 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:49,204 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:49,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:49,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:49,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:49,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:49,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:49,206 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:49,206 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40dd429[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:49,206 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:49,206 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:49,207 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:49,207 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:49,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55340 deadline: 1681315009200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:49,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:49,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:49,208 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:49,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55306 deadline: 1681315009200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:49,208 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:49,208 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:49,208 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:49,208 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:49,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55260 deadline: 1681315009201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:49,208 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:49,208 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:49,210 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:49,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:49,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:49,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:49,210 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:49,210 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:49,210 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:49,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:49,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:49,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:49,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:49,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:49,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:49,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:49,212 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:49,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55322 deadline: 1681315009205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:49,212 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:49,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55336 deadline: 1681315009205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:49,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:49,212 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:49,212 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:49,212 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:49,212 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:49,213 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:49,213 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:49,215 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:49,215 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:49,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:49,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:49,215 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:49,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:49,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:49,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:49,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:49,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:49,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:49,216 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:49,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:49,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:49,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:49,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:49,217 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:49,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:49,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:49,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:49,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:49,219 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:49,219 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:49,219 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:49,219 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:49,220 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:49,220 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:49,220 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:49,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:49,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:49,219 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:49,220 DEBUG [Thread-3125] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:49,220 DEBUG [Thread-3125] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:49,220 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:49,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:49,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:49,220 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:49,220 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:49,220 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:49,220 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:49,234 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/B/2352bcafbea44bee821b6ac9592c3553 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/2352bcafbea44bee821b6ac9592c3553 2023-04-12 15:55:49,239 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4df381dfb8e2792d3aca65faf2673530/B of 4df381dfb8e2792d3aca65faf2673530 into 2352bcafbea44bee821b6ac9592c3553(size=12.6 K), total size for store is 36.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:55:49,239 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4df381dfb8e2792d3aca65faf2673530: 2023-04-12 15:55:49,239 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530., storeName=4df381dfb8e2792d3aca65faf2673530/B, priority=13, startTime=1681314948806; duration=0sec 2023-04-12 15:55:49,239 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-12 15:55:49,239 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 15:55:49,240 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 15:55:49,240 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1912): 4df381dfb8e2792d3aca65faf2673530/C is initiating minor compaction (all files) 2023-04-12 15:55:49,240 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4df381dfb8e2792d3aca65faf2673530/C in TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:49,240 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/5bffd9da81d64cf88e694fdcca264c95, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/18a9bb7e91414ccca206b47d2e80cd78, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/a685dbb2493249d997efedbed80d4b7e] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp, totalSize=36.3 K 2023-04-12 15:55:49,247 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 5bffd9da81d64cf88e694fdcca264c95, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=314, earliestPutTs=1681314947953 2023-04-12 15:55:49,247 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 18a9bb7e91414ccca206b47d2e80cd78, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=329, earliestPutTs=1681314948336 2023-04-12 15:55:49,248 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting a685dbb2493249d997efedbed80d4b7e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=339, earliestPutTs=1681314949022 2023-04-12 15:55:49,256 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] throttle.PressureAwareThroughputController(145): 4df381dfb8e2792d3aca65faf2673530#C#compaction#468 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:55:49,320 DEBUG [Thread-3125] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:49,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:49,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:49,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:49,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:49,324 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:49,324 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:49,324 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:49,324 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:49,324 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:49,324 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:49,325 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:49,325 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:49,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:49,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:49,326 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:49,326 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:49,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:49,326 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:49,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:49,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:49,327 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:49,327 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:49,327 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:49,327 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:49,327 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:49,327 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:49,327 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:49,327 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:49,327 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:49,328 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. started... 2023-04-12 15:55:49,328 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. as already flushing 2023-04-12 15:55:49,328 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:49,328 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:49,328 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:49,328 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:49,328 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:49,328 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:49,330 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:49,330 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:49,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:49,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:49,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:49,330 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:49,330 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:49,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:49,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:49,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:49,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:49,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:49,332 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:49,332 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:49,332 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d5e50a1[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:49,332 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:49,336 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:49,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:49,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:49,337 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:49,337 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:49,337 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:49,337 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:49,337 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:49,338 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:49,338 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:49,338 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:49,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:49,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:49,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:49,338 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:49,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:49,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:49,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:49,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:49,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:49,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:49,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:49,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:49,342 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:49,342 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:49,342 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:49,342 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:49,342 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:49,342 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:49,344 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:49,344 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:49,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:49,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:49,344 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:49,344 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:49,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:49,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:49,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:49,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:49,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:49,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:49,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:49,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:49,345 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:49,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:49,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:49,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:49,362 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:49,362 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:49,362 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:49,362 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:49,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:49,363 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:49,363 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:49,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:49,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:49,362 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:49,362 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:49,363 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:49,363 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:49,363 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:49,363 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:49,363 DEBUG [Thread-3125] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:49,363 DEBUG [Thread-3125] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:49,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:49,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:49,364 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:49,414 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:49,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55306 deadline: 1681315009409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:49,414 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:49,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55340 deadline: 1681315009409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:49,415 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:49,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55260 deadline: 1681315009409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:49,420 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:49,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55336 deadline: 1681315009413, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:49,420 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:49,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55322 deadline: 1681315009414, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:49,463 DEBUG [Thread-3125] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:49,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:49,465 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:49,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:49,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:49,467 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:49,467 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:49,467 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:49,467 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:49,467 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:49,467 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:49,468 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:49,469 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:49,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:49,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:49,469 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:49,469 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:49,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:49,469 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:49,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:49,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:49,470 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:49,470 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:49,470 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:49,470 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:49,470 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:49,470 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:49,470 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:49,470 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:49,470 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. started... 2023-04-12 15:55:49,470 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. as already flushing 2023-04-12 15:55:49,471 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:49,471 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:49,471 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:49,471 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:49,471 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:49,471 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:49,472 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:49,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:49,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:49,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:49,472 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:49,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:49,473 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:49,473 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:49,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:49,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:49,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:49,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:49,474 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:49,474 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:49,474 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:49,474 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58dc7eff[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:49,476 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:49,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:49,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:49,476 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:49,476 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:49,476 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:49,477 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:49,477 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:49,478 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:49,478 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:49,478 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:49,478 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:49,478 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:49,478 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:49,478 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:49,478 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:49,478 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:49,479 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:49,479 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:49,479 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:49,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:49,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:49,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:49,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:49,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:49,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:49,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:49,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:49,480 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:49,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:49,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:49,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:49,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:49,481 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:49,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:49,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:49,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:49,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:49,483 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:49,484 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:49,484 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:49,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:49,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:49,484 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:49,484 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:49,484 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:49,484 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:49,484 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:49,484 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:49,484 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:49,484 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:49,484 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:49,484 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:49,484 DEBUG [Thread-3125] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:49,484 DEBUG [Thread-3125] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:49,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:49,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:49,498 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=367 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/A/79f0807411784c70a0a7432cff4a6fe8 2023-04-12 15:55:49,508 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=367 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/B/a3147bec335e4006954fc4bbcb0defc1 2023-04-12 15:55:49,509 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/A/4591bbe1462d42afbdbf1beeff30a53b as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/4591bbe1462d42afbdbf1beeff30a53b 2023-04-12 15:55:49,517 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4df381dfb8e2792d3aca65faf2673530/A of 4df381dfb8e2792d3aca65faf2673530 into 4591bbe1462d42afbdbf1beeff30a53b(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:55:49,517 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4df381dfb8e2792d3aca65faf2673530: 2023-04-12 15:55:49,517 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530., storeName=4df381dfb8e2792d3aca65faf2673530/A, priority=13, startTime=1681314949075; duration=0sec 2023-04-12 15:55:49,517 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:55:49,517 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 15:55:49,518 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37240 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 15:55:49,518 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1912): 4df381dfb8e2792d3aca65faf2673530/B is initiating minor compaction (all files) 2023-04-12 15:55:49,518 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4df381dfb8e2792d3aca65faf2673530/B in TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:49,518 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/2352bcafbea44bee821b6ac9592c3553, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/fae4637c71464ba093f316363f9bf3f1, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/f80962a0689242fba1a75853d3a7ddbc] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp, totalSize=36.4 K 2023-04-12 15:55:49,519 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 2352bcafbea44bee821b6ac9592c3553, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=314, earliestPutTs=1681314947953 2023-04-12 15:55:49,519 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting fae4637c71464ba093f316363f9bf3f1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=329, earliestPutTs=1681314948336 2023-04-12 15:55:49,520 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting f80962a0689242fba1a75853d3a7ddbc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=339, earliestPutTs=1681314949022 2023-04-12 15:55:49,520 WARN [DataStreamer for file /user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/C/acceaec84a894f129dd3c752e71a28c9] hdfs.DataStreamer(982): Caught exception java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1257) at java.lang.Thread.join(Thread.java:1331) at org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:980) at org.apache.hadoop.hdfs.DataStreamer.endBlock(DataStreamer.java:630) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:807) 2023-04-12 15:55:49,522 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=55.91 KB at sequenceid=367 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/C/acceaec84a894f129dd3c752e71a28c9 2023-04-12 15:55:49,526 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/A/79f0807411784c70a0a7432cff4a6fe8 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/79f0807411784c70a0a7432cff4a6fe8 2023-04-12 15:55:49,527 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4df381dfb8e2792d3aca65faf2673530#B#compaction#471 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:55:49,531 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/79f0807411784c70a0a7432cff4a6fe8, entries=200, sequenceid=367, filesize=14.3 K 2023-04-12 15:55:49,532 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/B/a3147bec335e4006954fc4bbcb0defc1 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/a3147bec335e4006954fc4bbcb0defc1 2023-04-12 15:55:49,536 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/a3147bec335e4006954fc4bbcb0defc1, entries=150, sequenceid=367, filesize=11.9 K 2023-04-12 15:55:49,537 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/C/acceaec84a894f129dd3c752e71a28c9 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/acceaec84a894f129dd3c752e71a28c9 2023-04-12 15:55:49,541 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/acceaec84a894f129dd3c752e71a28c9, entries=150, sequenceid=367, filesize=11.9 K 2023-04-12 15:55:49,542 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~118.53 KB/121370, heapSize ~334.05 KB/342064, currentSize=60.38 KB/61830 for 4df381dfb8e2792d3aca65faf2673530 in 462ms, sequenceid=367, compaction requested=false 2023-04-12 15:55:49,542 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4df381dfb8e2792d3aca65faf2673530: 2023-04-12 15:55:49,584 DEBUG [Thread-3125] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:49,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:49,587 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:49,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:49,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:49,588 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:49,588 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:49,588 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:49,588 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:49,588 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:49,588 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:49,590 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:49,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:49,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:49,590 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:49,591 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:49,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:49,591 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:49,591 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:49,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:49,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:49,591 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:49,592 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:49,592 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:49,592 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:49,592 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:49,592 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:49,592 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:49,592 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:49,592 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:49,592 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. started... 2023-04-12 15:55:49,592 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4df381dfb8e2792d3aca65faf2673530 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-12 15:55:49,593 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=A 2023-04-12 15:55:49,593 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:49,593 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=B 2023-04-12 15:55:49,593 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:49,593 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=C 2023-04-12 15:55:49,593 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:49,607 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=381 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/A/b8c1479fd7674e339d71f972981bc1d4 2023-04-12 15:55:49,665 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/C/a7bd8fb4177d4b46bcbb0c2c843c5802 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/a7bd8fb4177d4b46bcbb0c2c843c5802 2023-04-12 15:55:49,670 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4df381dfb8e2792d3aca65faf2673530/C of 4df381dfb8e2792d3aca65faf2673530 into a7bd8fb4177d4b46bcbb0c2c843c5802(size=12.6 K), total size for store is 24.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:55:49,670 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4df381dfb8e2792d3aca65faf2673530: 2023-04-12 15:55:49,670 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530., storeName=4df381dfb8e2792d3aca65faf2673530/C, priority=13, startTime=1681314949076; duration=0sec 2023-04-12 15:55:49,670 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:55:49,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on 4df381dfb8e2792d3aca65faf2673530 2023-04-12 15:55:49,720 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. as already flushing 2023-04-12 15:55:49,752 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:49,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55336 deadline: 1681315009743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:49,756 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:49,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55340 deadline: 1681315009745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:49,756 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:49,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681315009746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:49,756 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:49,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55306 deadline: 1681315009750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:49,757 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:49,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55260 deadline: 1681315009751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:49,857 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:49,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55336 deadline: 1681315009854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:49,861 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:49,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55340 deadline: 1681315009857, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:49,862 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:49,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681315009857, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:49,862 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:49,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55306 deadline: 1681315009857, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:49,862 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:49,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55260 deadline: 1681315009858, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:49,936 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/B/f4eb93e542a64a2cbf0f750369b00147 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/f4eb93e542a64a2cbf0f750369b00147 2023-04-12 15:55:49,940 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4df381dfb8e2792d3aca65faf2673530/B of 4df381dfb8e2792d3aca65faf2673530 into f4eb93e542a64a2cbf0f750369b00147(size=12.7 K), total size for store is 24.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:55:49,940 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4df381dfb8e2792d3aca65faf2673530: 2023-04-12 15:55:49,940 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530., storeName=4df381dfb8e2792d3aca65faf2673530/B, priority=13, startTime=1681314949075; duration=0sec 2023-04-12 15:55:49,941 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:55:50,021 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=381 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/B/482f0f11ea71403f846a451890f6cdeb 2023-04-12 15:55:50,064 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:50,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55336 deadline: 1681315010060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:50,065 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:50,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55340 deadline: 1681315010062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:50,065 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:50,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681315010063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:50,065 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:50,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55260 deadline: 1681315010063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:50,070 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:50,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55306 deadline: 1681315010064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:50,367 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:50,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55336 deadline: 1681315010366, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:50,367 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:50,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681315010366, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:50,371 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:50,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55340 deadline: 1681315010367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:50,371 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:50,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55260 deadline: 1681315010368, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:50,376 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:50,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55306 deadline: 1681315010373, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:50,432 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=381 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/C/3604312e5f944485bace7fd7977a9c96 2023-04-12 15:55:50,436 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/A/b8c1479fd7674e339d71f972981bc1d4 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/b8c1479fd7674e339d71f972981bc1d4 2023-04-12 15:55:50,439 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/b8c1479fd7674e339d71f972981bc1d4, entries=150, sequenceid=381, filesize=11.9 K 2023-04-12 15:55:50,440 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/B/482f0f11ea71403f846a451890f6cdeb as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/482f0f11ea71403f846a451890f6cdeb 2023-04-12 15:55:50,443 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/482f0f11ea71403f846a451890f6cdeb, entries=150, sequenceid=381, filesize=11.9 K 2023-04-12 15:55:50,444 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/C/3604312e5f944485bace7fd7977a9c96 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/3604312e5f944485bace7fd7977a9c96 2023-04-12 15:55:50,448 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/3604312e5f944485bace7fd7977a9c96, entries=150, sequenceid=381, filesize=11.9 K 2023-04-12 15:55:50,449 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 4df381dfb8e2792d3aca65faf2673530 in 857ms, sequenceid=381, compaction requested=true 2023-04-12 15:55:50,449 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4df381dfb8e2792d3aca65faf2673530: 2023-04-12 15:55:50,449 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:50,449 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:50,449 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:50,449 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:50,449 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:50,449 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:50,451 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:50,451 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:50,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:50,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:50,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:50,451 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:50,451 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:50,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:50,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:50,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:50,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:50,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:50,453 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:50,453 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43f1a928[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:50,453 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:50,453 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:50,454 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:50,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:50,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:50,454 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:50,454 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:50,454 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:50,454 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:50,454 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:50,455 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:50,455 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:50,455 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:50,455 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:50,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:50,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:50,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:50,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:50,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:50,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:50,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:50,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:50,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:50,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:50,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:50,458 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:50,458 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:50,458 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:50,458 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:50,458 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:50,458 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:50,460 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:50,460 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:50,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:50,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:50,461 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:50,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:50,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:50,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:50,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:50,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:50,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:50,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:50,461 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:50,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:50,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:50,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:50,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:50,462 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:50,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:50,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:50,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:50,465 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:50,465 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:50,465 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:50,465 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:50,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:50,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:50,465 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:50,465 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:50,465 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:50,465 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:50,465 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:50,465 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:50,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:50,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:50,465 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:50,466 DEBUG [Thread-3125] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:50,465 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:50,466 DEBUG [Thread-3125] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:50,566 DEBUG [Thread-3125] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:50,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:50,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:50,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:50,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:50,569 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:50,569 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:50,569 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:50,569 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:50,569 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:50,570 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:50,571 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:50,571 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:50,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:50,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:50,571 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:50,571 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:50,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:50,572 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:50,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:50,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:50,572 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:50,573 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:50,573 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:50,573 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:50,573 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:50,573 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:50,573 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:50,573 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:50,573 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:50,573 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. started... 2023-04-12 15:55:50,574 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4df381dfb8e2792d3aca65faf2673530 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-12 15:55:50,574 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=A 2023-04-12 15:55:50,574 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:50,574 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=B 2023-04-12 15:55:50,574 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:50,574 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=C 2023-04-12 15:55:50,574 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:50,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on 4df381dfb8e2792d3aca65faf2673530 2023-04-12 15:55:50,874 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. as already flushing 2023-04-12 15:55:50,886 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:50,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55336 deadline: 1681315010881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:50,886 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:50,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55340 deadline: 1681315010883, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:50,889 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:50,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55260 deadline: 1681315010883, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:50,889 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:50,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55306 deadline: 1681315010885, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:50,891 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:50,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681315010886, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:50,985 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=403 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/A/3868f81de34644519cbd3dc64f55a7aa 2023-04-12 15:55:50,990 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:50,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55336 deadline: 1681315010988, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:50,990 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:50,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55340 deadline: 1681315010988, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:50,997 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:50,997 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:50,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55260 deadline: 1681315010996, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:50,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681315010996, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:50,998 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:50,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55306 deadline: 1681315010996, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:51,200 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:51,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55340 deadline: 1681315011191, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:51,200 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:51,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55336 deadline: 1681315011192, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:51,202 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:51,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681315011198, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:51,202 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:51,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55260 deadline: 1681315011199, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:51,203 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:51,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55306 deadline: 1681315011200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:51,396 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=403 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/B/7dc6237b6f8a4a3199753193f1163ad3 2023-04-12 15:55:51,507 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:51,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55336 deadline: 1681315011502, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:51,507 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:51,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55340 deadline: 1681315011503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:51,507 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:51,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55260 deadline: 1681315011503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:51,508 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:51,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681315011504, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:51,509 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:51,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55306 deadline: 1681315011505, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:51,814 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=403 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/C/19902c3a096a4c17857583079aeb6ee4 2023-04-12 15:55:51,818 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/A/3868f81de34644519cbd3dc64f55a7aa as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/3868f81de34644519cbd3dc64f55a7aa 2023-04-12 15:55:51,822 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/3868f81de34644519cbd3dc64f55a7aa, entries=150, sequenceid=403, filesize=11.9 K 2023-04-12 15:55:51,823 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/B/7dc6237b6f8a4a3199753193f1163ad3 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/7dc6237b6f8a4a3199753193f1163ad3 2023-04-12 15:55:51,826 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/7dc6237b6f8a4a3199753193f1163ad3, entries=150, sequenceid=403, filesize=11.9 K 2023-04-12 15:55:51,827 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/C/19902c3a096a4c17857583079aeb6ee4 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/19902c3a096a4c17857583079aeb6ee4 2023-04-12 15:55:51,831 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/19902c3a096a4c17857583079aeb6ee4, entries=150, sequenceid=403, filesize=11.9 K 2023-04-12 15:55:51,831 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 4df381dfb8e2792d3aca65faf2673530 in 1257ms, sequenceid=403, compaction requested=true 2023-04-12 15:55:51,831 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4df381dfb8e2792d3aca65faf2673530: 2023-04-12 15:55:51,831 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:51,832 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:51,832 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:51,832 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:51,836 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:51,836 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:51,837 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:51,837 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:51,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:51,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:51,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:51,838 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:51,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:51,838 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:51,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:51,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:51,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:51,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:51,839 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:51,839 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@293791e3[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:51,839 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:51,839 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:51,845 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:51,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:51,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:51,845 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:51,845 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:51,845 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:51,845 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:51,845 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:51,847 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:51,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:51,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:51,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:51,847 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:51,847 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:51,847 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:51,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:51,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:51,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:51,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:51,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:51,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:51,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:51,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:51,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:51,849 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:51,849 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:51,849 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:51,849 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:51,849 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:51,851 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:51,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:51,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:51,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:51,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:51,851 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:51,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:51,851 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:51,851 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:51,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:51,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:51,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:51,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:51,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:51,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:51,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:51,853 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:51,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:51,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:51,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:51,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:51,859 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:51,859 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:51,860 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:51,860 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:51,860 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:51,860 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:51,860 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:51,860 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:51,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:51,860 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:51,860 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:51,860 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:51,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:51,860 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:51,860 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:51,860 DEBUG [Thread-3125] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:51,861 DEBUG [Thread-3125] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:51,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:51,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:51,961 DEBUG [Thread-3125] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:51,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:51,963 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:51,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:51,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:51,964 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:51,964 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:51,964 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:51,964 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:51,964 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:51,965 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:51,966 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:51,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:51,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:51,966 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:51,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:51,967 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:51,967 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:51,967 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:51,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:51,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:51,967 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:51,968 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:51,968 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:51,968 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:51,968 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:51,968 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:51,968 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:51,968 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:51,968 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:51,968 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. started... 2023-04-12 15:55:51,968 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4df381dfb8e2792d3aca65faf2673530 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-12 15:55:51,969 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=A 2023-04-12 15:55:51,969 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:51,969 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=B 2023-04-12 15:55:51,969 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:51,969 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=C 2023-04-12 15:55:51,969 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:51,976 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=415 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/A/d86e9d9a610a4366bf30c2ac9e64e0e0 2023-04-12 15:55:51,985 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=415 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/B/14783d764e7c4fc38554d605ee98a205 2023-04-12 15:55:51,997 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=415 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/C/45e685c74f0948ac80026e409c72a3fe 2023-04-12 15:55:52,001 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/A/d86e9d9a610a4366bf30c2ac9e64e0e0 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/d86e9d9a610a4366bf30c2ac9e64e0e0 2023-04-12 15:55:52,004 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/d86e9d9a610a4366bf30c2ac9e64e0e0, entries=150, sequenceid=415, filesize=11.9 K 2023-04-12 15:55:52,005 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/B/14783d764e7c4fc38554d605ee98a205 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/14783d764e7c4fc38554d605ee98a205 2023-04-12 15:55:52,009 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/14783d764e7c4fc38554d605ee98a205, entries=150, sequenceid=415, filesize=11.9 K 2023-04-12 15:55:52,009 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/C/45e685c74f0948ac80026e409c72a3fe as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/45e685c74f0948ac80026e409c72a3fe 2023-04-12 15:55:52,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on 4df381dfb8e2792d3aca65faf2673530 2023-04-12 15:55:52,014 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. as already flushing 2023-04-12 15:55:52,016 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/45e685c74f0948ac80026e409c72a3fe, entries=150, sequenceid=415, filesize=11.9 K 2023-04-12 15:55:52,017 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=20.13 KB/20610 for 4df381dfb8e2792d3aca65faf2673530 in 48ms, sequenceid=415, compaction requested=true 2023-04-12 15:55:52,017 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4df381dfb8e2792d3aca65faf2673530: 2023-04-12 15:55:52,017 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:52,017 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:52,017 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:52,017 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,017 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:52,017 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:52,020 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,020 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:52,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:52,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:52,021 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:52,021 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:52,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:52,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:52,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:52,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:52,022 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:52,022 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1653173c[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:52,022 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:52,022 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:52,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on 4df381dfb8e2792d3aca65faf2673530 2023-04-12 15:55:52,024 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4df381dfb8e2792d3aca65faf2673530 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-12 15:55:52,025 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=A 2023-04-12 15:55:52,025 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:52,025 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=B 2023-04-12 15:55:52,025 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:52,025 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=C 2023-04-12 15:55:52,025 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:52,025 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:52,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:52,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:52,025 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:52,025 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:52,025 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:52,025 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,025 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:52,027 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:52,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:52,027 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:52,028 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:52,028 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:52,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:52,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:52,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:52,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:52,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:52,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:52,030 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:52,030 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:52,030 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:52,030 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:52,030 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:52,030 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:52,033 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,034 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,034 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:52,035 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:52,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:52,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:52,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:52,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:52,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:52,035 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:52,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:52,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:52,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:52,037 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:52,037 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:52,037 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:52,037 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:52,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:52,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:52,037 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:52,037 DEBUG [Thread-3125] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:52,037 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:52,037 DEBUG [Thread-3125] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:52,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:52,037 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:52,037 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,037 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:52,038 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:52,038 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:52,038 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:52,057 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:52,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55340 deadline: 1681315012049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,068 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:52,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55260 deadline: 1681315012055, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,068 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:52,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55306 deadline: 1681315012055, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,068 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:52,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55336 deadline: 1681315012056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,069 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:52,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681315012057, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,137 DEBUG [Thread-3125] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:52,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:52,140 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:52,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:52,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:52,141 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:52,141 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:52,141 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:52,141 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:52,143 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:52,143 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:52,146 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:52,146 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:52,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:52,146 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,146 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:52,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:52,147 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:52,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:52,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:52,147 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:52,147 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:52,147 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:52,147 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:52,147 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:52,147 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:52,148 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:52,148 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:52,148 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:52,148 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. started... 2023-04-12 15:55:52,148 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. as already flushing 2023-04-12 15:55:52,148 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:52,148 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:52,148 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:52,148 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,148 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:52,148 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:52,150 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,150 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:52,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:52,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:52,150 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:52,150 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:52,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:52,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:52,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:52,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:52,153 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:52,153 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d43a88f[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:52,153 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:52,153 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:52,154 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:52,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:52,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:52,155 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:52,155 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:52,155 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:52,155 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,155 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:52,156 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:52,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:52,156 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:52,156 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:52,156 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:52,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:52,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:52,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:52,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:52,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:52,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:52,160 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:52,160 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:52,160 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:52,160 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:52,160 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:52,160 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:52,162 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:52,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55340 deadline: 1681315012158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,163 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:52,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:52,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:52,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:52,163 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:52,163 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:52,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:52,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:52,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:52,163 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:52,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:52,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:52,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:52,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:52,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:52,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,165 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:52,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:52,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,167 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:52,167 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:52,167 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,167 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:52,167 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:52,167 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:52,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:52,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:52,167 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:52,167 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:52,167 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:52,167 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,167 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:52,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:52,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:52,167 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:52,167 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:52,168 DEBUG [Thread-3125] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:52,168 DEBUG [Thread-3125] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:52,175 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:52,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55336 deadline: 1681315012170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,175 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:52,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55260 deadline: 1681315012170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,176 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:52,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681315012170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,176 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:52,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55306 deadline: 1681315012170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,268 DEBUG [Thread-3125] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:52,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:52,270 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:52,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:52,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:52,271 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:52,271 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:52,271 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:52,271 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:52,272 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:52,272 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:52,273 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:52,273 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:52,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:52,273 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,273 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:52,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:52,274 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:52,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:52,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:52,274 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:52,274 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:52,274 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:52,274 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:52,275 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:52,275 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:52,275 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:52,275 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:52,275 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:52,275 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. started... 2023-04-12 15:55:52,275 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. as already flushing 2023-04-12 15:55:52,275 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:52,275 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:52,275 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:52,275 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,275 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:52,275 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:52,276 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,277 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:52,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:52,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:52,277 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:52,277 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:52,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:52,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:52,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:52,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:52,279 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:52,279 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6335ece6[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:52,280 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:52,280 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:52,287 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:52,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:52,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:52,287 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:52,287 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:52,287 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:52,287 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,287 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:52,289 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:52,289 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:52,289 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,289 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:52,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:52,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:52,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:52,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:52,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:52,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:52,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:52,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:52,291 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:52,291 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:52,291 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:52,291 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:52,291 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:52,291 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:52,293 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:52,293 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:52,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:52,293 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:52,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:52,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:52,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:52,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:52,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:52,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:52,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:52,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:52,294 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:52,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:52,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:52,295 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:52,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:52,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,299 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,299 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:52,299 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:52,299 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,299 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:52,299 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:52,299 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:52,299 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:52,299 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:52,299 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:52,299 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:52,299 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:52,299 DEBUG [Thread-3125] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:52,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:52,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:52,299 DEBUG [Thread-3125] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:52,299 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:52,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:52,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:52,377 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:52,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55340 deadline: 1681315012364, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,383 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:52,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55336 deadline: 1681315012377, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,383 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:52,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55260 deadline: 1681315012377, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,383 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:52,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55306 deadline: 1681315012378, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,383 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:52,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681315012378, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,400 DEBUG [Thread-3125] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:52,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:52,402 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:52,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:52,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:52,403 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:52,403 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:52,403 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:52,403 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:52,404 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:52,404 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:52,406 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,406 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:52,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:52,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:52,406 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,406 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:52,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:52,407 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:52,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:52,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:52,407 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:52,407 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:52,407 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:52,408 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:52,408 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:52,408 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:52,408 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:52,408 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:52,408 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. started... 2023-04-12 15:55:52,408 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. as already flushing 2023-04-12 15:55:52,408 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:52,408 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:52,408 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:52,408 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,408 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:52,408 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:52,415 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,415 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:52,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:52,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:52,416 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:52,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:52,416 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:52,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:52,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:52,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:52,417 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:52,417 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7618d1ea[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:52,417 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:52,417 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:52,418 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:52,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:52,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:52,419 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:52,419 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:52,419 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,419 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:52,419 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:52,420 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:52,420 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,420 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:52,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:52,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:52,420 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:52,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:52,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:52,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:52,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:52,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:52,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:52,423 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:52,423 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:52,423 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:52,423 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:52,423 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:52,423 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:52,426 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:52,426 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:52,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:52,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:52,426 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:52,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:52,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:52,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:52,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:52,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:52,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:52,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:52,426 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:52,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:52,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:52,427 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:52,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:52,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,429 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,430 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:52,430 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:52,430 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,430 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:52,430 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:52,430 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:52,430 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:52,430 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:52,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:52,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:52,430 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:52,430 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:52,430 DEBUG [Thread-3125] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:52,430 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:52,430 DEBUG [Thread-3125] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:52,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:52,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:52,432 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=427 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/A/f5a4cc23d1944422b0b06b4041c0a1e9 2023-04-12 15:55:52,530 DEBUG [Thread-3125] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:52,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:52,533 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:52,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:52,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:52,534 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:52,534 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:52,534 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:52,534 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:52,535 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:52,535 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:52,537 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,537 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:52,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:52,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:52,537 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,537 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:52,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:52,538 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:52,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:52,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:52,538 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:52,538 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:52,539 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:52,539 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:52,539 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:52,539 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:52,539 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:52,539 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:52,539 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:52,539 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. started... 2023-04-12 15:55:52,539 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. as already flushing 2023-04-12 15:55:52,539 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:52,539 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:52,539 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:52,539 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,539 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:52,539 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:52,541 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,541 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:52,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:52,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:52,542 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:52,542 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:52,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:52,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:52,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:52,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:52,543 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:52,543 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78fcd2e2[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:52,543 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:52,543 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:52,545 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:52,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:52,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:52,545 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:52,545 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:52,545 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:52,545 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,545 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:52,547 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,547 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:52,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:52,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:52,547 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:52,548 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:52,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:52,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:52,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:52,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:52,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:52,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:52,550 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:52,550 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:52,550 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:52,550 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:52,550 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:52,550 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:52,551 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:52,551 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:52,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:52,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:52,551 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:52,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:52,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:52,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:52,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:52,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:52,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:52,552 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:52,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:52,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:52,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:52,554 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:52,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:52,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,557 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,557 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:52,557 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:52,557 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,557 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:52,557 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:52,557 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:52,557 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:52,557 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:52,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:52,557 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:52,557 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:52,557 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:52,557 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:52,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:52,558 DEBUG [Thread-3125] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:52,558 DEBUG [Thread-3125] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:52,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:52,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:52,658 DEBUG [Thread-3125] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:52,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:52,660 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:52,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:52,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:52,661 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:52,662 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:52,662 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:52,662 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:52,662 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:52,662 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:52,664 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:52,664 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:52,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:52,664 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,664 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:52,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:52,665 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:52,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:52,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:52,665 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:52,665 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:52,666 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:52,666 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:52,666 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:52,666 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:52,666 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:52,666 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:52,666 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:52,666 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. started... 2023-04-12 15:55:52,666 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. as already flushing 2023-04-12 15:55:52,666 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:52,666 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:52,666 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:52,666 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,666 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:52,666 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:52,668 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,668 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:52,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:52,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:52,668 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:52,668 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:52,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:52,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:52,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:52,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:52,670 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:52,670 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@484b1aa2[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:52,670 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:52,670 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:52,671 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:52,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:52,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:52,671 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:52,671 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:52,671 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:52,671 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,671 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:52,673 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,673 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:52,673 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:52,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:52,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:52,673 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:52,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:52,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:52,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:52,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:52,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:52,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:52,675 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:52,675 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:52,675 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:52,675 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:52,675 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:52,675 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:52,679 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:52,679 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:52,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:52,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:52,679 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:52,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:52,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:52,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:52,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:52,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:52,680 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:52,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:52,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:52,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:52,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:52,681 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:52,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:52,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,683 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:52,683 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:52,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55340 deadline: 1681315012678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:52,684 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,684 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:52,684 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:52,684 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,684 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:52,684 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:52,684 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:52,685 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:52,684 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:52,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:52,684 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:52,685 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:52,684 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:52,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:52,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:52,686 DEBUG [Thread-3125] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:52,686 DEBUG [Thread-3125] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:52,687 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:52,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55336 deadline: 1681315012684, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,688 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:52,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681315012684, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,688 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:52,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55260 deadline: 1681315012685, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,688 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:52,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55306 deadline: 1681315012685, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,786 DEBUG [Thread-3125] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:52,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:52,788 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:52,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:52,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:52,789 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:52,789 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:52,789 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:52,789 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:52,790 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:52,790 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:52,791 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,791 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,791 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:52,792 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:52,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:52,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:52,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:52,793 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:52,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:52,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:52,794 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:52,794 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:52,794 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:52,794 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:52,794 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:52,794 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:52,794 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:52,794 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:52,794 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:52,794 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. started... 2023-04-12 15:55:52,795 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. as already flushing 2023-04-12 15:55:52,795 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:52,795 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:52,795 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:52,795 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,795 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:52,795 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:52,796 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,796 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:52,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:52,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:52,797 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:52,797 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:52,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:52,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:52,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:52,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:52,798 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:52,798 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d59d91[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:52,798 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:52,798 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:52,808 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:52,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:52,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:52,808 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:52,808 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,808 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:52,808 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:52,808 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:52,810 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,810 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:52,810 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:52,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:52,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:52,810 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:52,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:52,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:52,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:52,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:52,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:52,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:52,813 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:52,813 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:52,813 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:52,813 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:52,813 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:52,813 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:52,815 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:52,815 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:52,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:52,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:52,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:52,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:52,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:52,815 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:52,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:52,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:52,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:52,815 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:52,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:52,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:52,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:52,817 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:52,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:52,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,819 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,819 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:52,819 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:52,819 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,819 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:52,819 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:52,819 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:52,820 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:52,819 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:52,820 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:52,819 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:52,820 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:52,820 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:52,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:52,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:52,820 DEBUG [Thread-3125] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:52,820 DEBUG [Thread-3125] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:52,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:52,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:52,841 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=427 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/B/32e32955ee5d4841ab3bb471d0d971d5 2023-04-12 15:55:52,851 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=427 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/C/6303bac5909042efa692f06a2cbef356 2023-04-12 15:55:52,856 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/A/f5a4cc23d1944422b0b06b4041c0a1e9 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/f5a4cc23d1944422b0b06b4041c0a1e9 2023-04-12 15:55:52,859 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/f5a4cc23d1944422b0b06b4041c0a1e9, entries=150, sequenceid=427, filesize=11.9 K 2023-04-12 15:55:52,860 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/B/32e32955ee5d4841ab3bb471d0d971d5 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/32e32955ee5d4841ab3bb471d0d971d5 2023-04-12 15:55:52,863 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/32e32955ee5d4841ab3bb471d0d971d5, entries=100, sequenceid=427, filesize=9.5 K 2023-04-12 15:55:52,864 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/C/6303bac5909042efa692f06a2cbef356 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/6303bac5909042efa692f06a2cbef356 2023-04-12 15:55:52,875 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/6303bac5909042efa692f06a2cbef356, entries=100, sequenceid=427, filesize=9.5 K 2023-04-12 15:55:52,876 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 4df381dfb8e2792d3aca65faf2673530 in 852ms, sequenceid=427, compaction requested=true 2023-04-12 15:55:52,876 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4df381dfb8e2792d3aca65faf2673530: 2023-04-12 15:55:52,876 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:55:52,876 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-12 15:55:52,876 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-12 15:55:52,876 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:55:52,877 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:55:52,880 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71456 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-12 15:55:52,880 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1912): 4df381dfb8e2792d3aca65faf2673530/B is initiating minor compaction (all files) 2023-04-12 15:55:52,880 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4df381dfb8e2792d3aca65faf2673530/B in TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:52,880 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/f4eb93e542a64a2cbf0f750369b00147, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/a3147bec335e4006954fc4bbcb0defc1, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/482f0f11ea71403f846a451890f6cdeb, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/7dc6237b6f8a4a3199753193f1163ad3, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/14783d764e7c4fc38554d605ee98a205, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/32e32955ee5d4841ab3bb471d0d971d5] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp, totalSize=69.8 K 2023-04-12 15:55:52,880 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 76336 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-12 15:55:52,880 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1912): 4df381dfb8e2792d3aca65faf2673530/A is initiating minor compaction (all files) 2023-04-12 15:55:52,880 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4df381dfb8e2792d3aca65faf2673530/A in TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:52,881 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/4591bbe1462d42afbdbf1beeff30a53b, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/79f0807411784c70a0a7432cff4a6fe8, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/b8c1479fd7674e339d71f972981bc1d4, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/3868f81de34644519cbd3dc64f55a7aa, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/d86e9d9a610a4366bf30c2ac9e64e0e0, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/f5a4cc23d1944422b0b06b4041c0a1e9] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp, totalSize=74.5 K 2023-04-12 15:55:52,882 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting f4eb93e542a64a2cbf0f750369b00147, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=339, earliestPutTs=1681314949022 2023-04-12 15:55:52,882 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 4591bbe1462d42afbdbf1beeff30a53b, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=339, earliestPutTs=1681314949022 2023-04-12 15:55:52,882 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting a3147bec335e4006954fc4bbcb0defc1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1681314949063 2023-04-12 15:55:52,882 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 79f0807411784c70a0a7432cff4a6fe8, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1681314949062 2023-04-12 15:55:52,882 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 482f0f11ea71403f846a451890f6cdeb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=381, earliestPutTs=1681314949088 2023-04-12 15:55:52,883 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting b8c1479fd7674e339d71f972981bc1d4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=381, earliestPutTs=1681314949088 2023-04-12 15:55:52,883 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 7dc6237b6f8a4a3199753193f1163ad3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=403, earliestPutTs=1681314949744 2023-04-12 15:55:52,883 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 3868f81de34644519cbd3dc64f55a7aa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=403, earliestPutTs=1681314949744 2023-04-12 15:55:52,883 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 14783d764e7c4fc38554d605ee98a205, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=415, earliestPutTs=1681314950882 2023-04-12 15:55:52,883 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting d86e9d9a610a4366bf30c2ac9e64e0e0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=415, earliestPutTs=1681314950882 2023-04-12 15:55:52,883 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 32e32955ee5d4841ab3bb471d0d971d5, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=427, earliestPutTs=1681314952013 2023-04-12 15:55:52,884 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting f5a4cc23d1944422b0b06b4041c0a1e9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=427, earliestPutTs=1681314952013 2023-04-12 15:55:52,896 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4df381dfb8e2792d3aca65faf2673530#A#compaction#484 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:55:52,900 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] throttle.PressureAwareThroughputController(145): 4df381dfb8e2792d3aca65faf2673530#B#compaction#485 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:55:52,912 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/A/99a9ad6cfbab4e659f38d9d9b9b3d270 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/99a9ad6cfbab4e659f38d9d9b9b3d270 2023-04-12 15:55:52,916 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 4df381dfb8e2792d3aca65faf2673530/A of 4df381dfb8e2792d3aca65faf2673530 into 99a9ad6cfbab4e659f38d9d9b9b3d270(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:55:52,916 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4df381dfb8e2792d3aca65faf2673530: 2023-04-12 15:55:52,916 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530., storeName=4df381dfb8e2792d3aca65faf2673530/A, priority=10, startTime=1681314952876; duration=0sec 2023-04-12 15:55:52,916 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:55:52,916 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-12 15:55:52,918 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71422 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-12 15:55:52,918 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1912): 4df381dfb8e2792d3aca65faf2673530/C is initiating minor compaction (all files) 2023-04-12 15:55:52,918 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4df381dfb8e2792d3aca65faf2673530/C in TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:52,918 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/a7bd8fb4177d4b46bcbb0c2c843c5802, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/acceaec84a894f129dd3c752e71a28c9, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/3604312e5f944485bace7fd7977a9c96, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/19902c3a096a4c17857583079aeb6ee4, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/45e685c74f0948ac80026e409c72a3fe, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/6303bac5909042efa692f06a2cbef356] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp, totalSize=69.7 K 2023-04-12 15:55:52,918 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting a7bd8fb4177d4b46bcbb0c2c843c5802, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=339, earliestPutTs=1681314949022 2023-04-12 15:55:52,918 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting acceaec84a894f129dd3c752e71a28c9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1681314949063 2023-04-12 15:55:52,919 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 3604312e5f944485bace7fd7977a9c96, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=381, earliestPutTs=1681314949088 2023-04-12 15:55:52,919 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 19902c3a096a4c17857583079aeb6ee4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=403, earliestPutTs=1681314949744 2023-04-12 15:55:52,919 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 45e685c74f0948ac80026e409c72a3fe, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=415, earliestPutTs=1681314950882 2023-04-12 15:55:52,919 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 6303bac5909042efa692f06a2cbef356, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=427, earliestPutTs=1681314952013 2023-04-12 15:55:52,920 DEBUG [Thread-3125] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:52,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:52,922 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:52,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:52,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:52,926 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:52,926 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:52,926 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:52,926 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:52,926 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:52,926 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:52,928 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:52,928 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:52,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:52,928 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:52,928 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:52,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:52,928 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4df381dfb8e2792d3aca65faf2673530#C#compaction#486 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:55:52,928 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:52,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:52,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:52,929 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:52,929 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:52,929 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:52,929 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:52,929 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:52,929 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:52,932 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:52,932 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:52,932 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:52,932 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. started... 2023-04-12 15:55:52,932 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4df381dfb8e2792d3aca65faf2673530 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-12 15:55:52,932 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=A 2023-04-12 15:55:52,932 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:52,932 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=B 2023-04-12 15:55:52,932 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:52,932 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=C 2023-04-12 15:55:52,933 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:52,954 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=448 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/A/ec600f2c00ff478ba737ed5f2cc7479a 2023-04-12 15:55:52,959 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/C/6e7333c2a545494d8452781dab218e1f as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/6e7333c2a545494d8452781dab218e1f 2023-04-12 15:55:52,969 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 4df381dfb8e2792d3aca65faf2673530/C of 4df381dfb8e2792d3aca65faf2673530 into 6e7333c2a545494d8452781dab218e1f(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:55:52,969 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4df381dfb8e2792d3aca65faf2673530: 2023-04-12 15:55:52,969 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530., storeName=4df381dfb8e2792d3aca65faf2673530/C, priority=10, startTime=1681314952876; duration=0sec 2023-04-12 15:55:52,969 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:55:52,985 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=448 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/B/035b5dfb907c47069b2ea091cf343ff0 2023-04-12 15:55:53,006 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=448 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/C/d02c7ea22ce34295b782128619720dae 2023-04-12 15:55:53,018 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/A/ec600f2c00ff478ba737ed5f2cc7479a as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/ec600f2c00ff478ba737ed5f2cc7479a 2023-04-12 15:55:53,024 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/ec600f2c00ff478ba737ed5f2cc7479a, entries=150, sequenceid=448, filesize=11.9 K 2023-04-12 15:55:53,025 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/B/035b5dfb907c47069b2ea091cf343ff0 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/035b5dfb907c47069b2ea091cf343ff0 2023-04-12 15:55:53,030 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/035b5dfb907c47069b2ea091cf343ff0, entries=150, sequenceid=448, filesize=11.9 K 2023-04-12 15:55:53,031 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/C/d02c7ea22ce34295b782128619720dae as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/d02c7ea22ce34295b782128619720dae 2023-04-12 15:55:53,036 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/d02c7ea22ce34295b782128619720dae, entries=150, sequenceid=448, filesize=11.9 K 2023-04-12 15:55:53,037 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=0 B/0 for 4df381dfb8e2792d3aca65faf2673530 in 105ms, sequenceid=448, compaction requested=false 2023-04-12 15:55:53,037 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4df381dfb8e2792d3aca65faf2673530: 2023-04-12 15:55:53,038 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:53,038 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:53,038 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:53,038 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,038 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:53,038 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:53,039 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:53,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:53,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:53,043 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:53,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:53,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:53,044 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:53,044 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:53,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:53,045 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:53,045 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34729b60[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:53,045 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:53,045 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:53,047 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:53,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:53,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:53,047 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:53,047 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:53,047 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:53,050 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,050 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:53,050 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:53,050 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:53,050 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:53,050 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:53,050 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:53,050 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:53,051 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:53,051 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:53,051 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:53,052 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:53,052 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:53,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:53,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:53,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:53,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:53,053 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:53,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:53,053 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:53,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:53,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:53,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:53,056 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:53,058 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,058 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:53,058 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:53,058 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,058 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:53,059 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:53,059 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:53,059 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:53,058 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:53,059 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:53,059 DEBUG [Thread-3125] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:53,059 DEBUG [Thread-3125] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:53,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:53,059 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:53,059 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:53,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:53,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:53,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:53,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:53,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:53,063 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:53,159 DEBUG [Thread-3125] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:53,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:53,162 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:53,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:53,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:53,163 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:53,163 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:53,163 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:53,163 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:53,163 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:53,164 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:53,165 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:53,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:53,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:53,165 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:53,165 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,166 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:53,166 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:53,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:53,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:53,166 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:53,167 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:53,167 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:53,167 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:53,167 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:53,167 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:53,168 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:53,168 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:53,168 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:53,168 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. started... 2023-04-12 15:55:53,168 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4df381dfb8e2792d3aca65faf2673530: 2023-04-12 15:55:53,168 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:53,168 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:53,168 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:53,168 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,168 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:53,168 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:53,170 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,170 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:53,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:53,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:53,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:53,171 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:53,171 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:53,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:53,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:53,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:53,173 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:53,173 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a0724ce[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:53,173 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:53,173 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:53,174 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:53,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:53,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:53,174 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:53,174 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:53,175 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:53,175 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,175 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:53,176 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,176 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:53,176 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:53,176 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:53,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:53,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:53,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:53,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:53,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:53,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:53,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:53,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:53,180 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:53,180 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:53,181 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:53,181 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:53,181 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:53,181 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:53,184 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:53,184 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:53,184 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:53,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:53,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:53,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:53,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:53,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:53,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:53,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:53,184 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:53,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:53,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:53,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:53,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:53,186 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:53,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:53,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,192 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:53,192 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:53,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:53,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:53,192 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,192 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:53,192 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:53,192 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:53,192 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:53,192 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:53,192 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,192 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:53,192 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:53,192 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:53,192 DEBUG [Thread-3125] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:53,193 DEBUG [Thread-3125] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:53,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:53,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:53,193 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:53,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on 4df381dfb8e2792d3aca65faf2673530 2023-04-12 15:55:53,197 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4df381dfb8e2792d3aca65faf2673530 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-12 15:55:53,198 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=A 2023-04-12 15:55:53,198 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:53,198 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=B 2023-04-12 15:55:53,198 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:53,198 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=C 2023-04-12 15:55:53,199 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:53,224 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:53,224 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:53,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55336 deadline: 1681315013218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681315013218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,225 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:53,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55340 deadline: 1681315013219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,225 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681314953216, max timestamp=1681314953223Num uniques 859; 2023-04-12 15:55:53,225 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681314953216, max timestamp=1681314953223Num uniques 859; 2023-04-12 15:55:53,225 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681314953216, max timestamp=1681314953223Num uniques 859; 2023-04-12 15:55:53,226 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-12 15:55:53,226 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-12 15:55:53,225 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-12 15:55:53,289 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:53,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55306 deadline: 1681315013282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,293 DEBUG [Thread-3125] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:53,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:53,295 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:53,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:53,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:53,296 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:53,296 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:53,296 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:53,296 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:53,296 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:53,296 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:53,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55260 deadline: 1681315013283, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,296 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:53,298 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,298 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:53,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:53,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:53,298 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,299 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:53,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:53,299 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:53,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:53,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:53,299 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:53,299 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:53,299 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:53,299 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:53,299 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:53,300 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:53,300 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:53,300 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:53,300 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:53,300 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. started... 2023-04-12 15:55:53,300 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. as already flushing 2023-04-12 15:55:53,300 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:53,300 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:53,300 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:53,300 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,300 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:53,300 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:53,302 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:53,302 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:53,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:53,302 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:53,302 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:53,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:53,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:53,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:53,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:53,304 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:53,304 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:53,304 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:53,304 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a89a526[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:53,305 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:53,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:53,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:53,305 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:53,305 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:53,305 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:53,305 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,305 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:53,312 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,312 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:53,312 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:53,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:53,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:53,312 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:53,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:53,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:53,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:53,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:53,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:53,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:53,315 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:53,315 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:53,315 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:53,315 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:53,315 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:53,315 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:53,317 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:53,317 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:53,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:53,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:53,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:53,317 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:53,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:53,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:53,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:53,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:53,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:53,318 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:53,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:53,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:53,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:53,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,319 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:53,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:53,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,322 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,322 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:53,322 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:53,322 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:53,322 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:53,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:53,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:53,322 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:53,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:53,322 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:53,322 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:53,322 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:53,322 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:53,322 DEBUG [Thread-3125] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:53,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:53,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:53,322 DEBUG [Thread-3125] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:53,330 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/B/61fec2329fdc4306ac0637dd065ea208 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/61fec2329fdc4306ac0637dd065ea208 2023-04-12 15:55:53,334 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 4df381dfb8e2792d3aca65faf2673530/B of 4df381dfb8e2792d3aca65faf2673530 into 61fec2329fdc4306ac0637dd065ea208(size=12.9 K), total size for store is 24.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:55:53,334 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4df381dfb8e2792d3aca65faf2673530: 2023-04-12 15:55:53,334 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530., storeName=4df381dfb8e2792d3aca65faf2673530/B, priority=10, startTime=1681314952876; duration=0sec 2023-04-12 15:55:53,334 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:55:53,336 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:53,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681315013326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,337 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:53,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55340 deadline: 1681315013327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,337 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:53,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55336 deadline: 1681315013327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,393 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:53,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55306 deadline: 1681315013390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,401 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:53,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55260 deadline: 1681315013397, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,422 DEBUG [Thread-3125] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:53,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:53,425 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:53,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:53,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:53,426 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:53,426 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:53,426 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:53,426 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:53,427 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:53,427 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:53,428 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:53,428 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:53,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:53,429 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,429 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:53,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:53,429 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:53,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:53,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:53,430 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:53,430 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:53,430 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:53,430 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:53,430 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:53,430 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:53,430 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:53,430 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:53,430 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:53,430 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. started... 2023-04-12 15:55:53,430 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. as already flushing 2023-04-12 15:55:53,430 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:53,431 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:53,431 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:53,431 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,431 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:53,431 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:53,433 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,433 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:53,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:53,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:53,433 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:53,433 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:53,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:53,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:53,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:53,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:53,435 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:53,435 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12989ef3[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:53,435 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:53,435 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:53,436 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:53,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:53,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:53,436 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:53,436 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:53,436 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:53,436 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,436 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:53,437 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:53,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:53,438 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:53,438 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:53,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:53,438 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:53,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:53,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:53,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:53,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:53,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:53,442 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:53,442 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:53,442 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:53,442 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:53,442 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:53,442 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:53,444 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:53,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:53,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:53,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:53,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:53,445 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,445 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:53,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:53,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:53,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:53,445 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:53,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:53,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:53,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:53,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:53,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:53,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,446 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:53,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:53,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,449 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:53,449 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:53,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:53,449 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:53,449 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:53,449 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:53,449 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:53,449 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:53,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:53,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:53,449 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:53,449 DEBUG [Thread-3125] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:53,449 DEBUG [Thread-3125] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:53,449 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,449 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:53,449 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:53,449 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:53,543 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:53,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681315013539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,543 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:53,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55340 deadline: 1681315013539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,543 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:53,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55336 deadline: 1681315013539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,549 DEBUG [Thread-3125] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:53,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:53,552 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:53,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:53,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:53,553 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:53,553 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:53,553 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:53,553 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:53,553 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:53,553 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:53,554 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:53,554 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:53,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:53,555 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,555 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:53,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:53,555 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:53,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:53,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:53,556 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:53,556 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:53,556 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:53,556 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:53,556 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:53,556 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:53,556 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:53,556 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:53,556 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:53,556 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. started... 2023-04-12 15:55:53,556 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. as already flushing 2023-04-12 15:55:53,556 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:53,557 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:53,557 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:53,557 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,557 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:53,557 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:53,558 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:53,558 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:53,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:53,559 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:53,559 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:53,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:53,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:53,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:53,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:53,560 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:53,560 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3211be5b[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:53,560 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:53,560 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:53,562 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:53,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:53,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:53,563 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:53,563 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:53,563 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:53,563 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,563 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:53,564 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:53,564 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,564 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:53,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:53,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:53,564 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:53,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:53,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:53,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:53,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:53,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:53,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:53,567 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:53,567 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:53,567 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:53,567 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:53,567 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:53,567 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:53,568 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:53,568 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:53,568 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:53,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:53,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:53,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:53,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:53,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:53,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:53,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:53,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:53,569 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:53,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:53,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:53,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:53,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,571 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:53,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:53,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,574 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:53,574 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,574 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:53,574 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:53,574 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:53,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:53,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:53,574 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:53,574 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:53,574 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:53,574 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,574 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:53,574 DEBUG [Thread-3125] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:53,574 DEBUG [Thread-3125] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:53,574 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:53,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:53,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:53,574 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:53,598 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:53,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55306 deadline: 1681315013595, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,605 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:53,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55260 deadline: 1681315013603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,606 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=461 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/A/c3b12e3a43b245b59c783ca2da5b739e 2023-04-12 15:55:53,618 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=461 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/B/9daf49a342ae46559cced96796e63846 2023-04-12 15:55:53,632 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=461 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/C/06b0d58b1a7b4e6c9e199aeada295bae 2023-04-12 15:55:53,639 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/A/c3b12e3a43b245b59c783ca2da5b739e as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/c3b12e3a43b245b59c783ca2da5b739e 2023-04-12 15:55:53,644 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/c3b12e3a43b245b59c783ca2da5b739e, entries=100, sequenceid=461, filesize=9.5 K 2023-04-12 15:55:53,645 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/B/9daf49a342ae46559cced96796e63846 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/9daf49a342ae46559cced96796e63846 2023-04-12 15:55:53,650 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/9daf49a342ae46559cced96796e63846, entries=100, sequenceid=461, filesize=9.5 K 2023-04-12 15:55:53,650 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/C/06b0d58b1a7b4e6c9e199aeada295bae as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/06b0d58b1a7b4e6c9e199aeada295bae 2023-04-12 15:55:53,653 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/06b0d58b1a7b4e6c9e199aeada295bae, entries=100, sequenceid=461, filesize=9.5 K 2023-04-12 15:55:53,653 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 4df381dfb8e2792d3aca65faf2673530 in 456ms, sequenceid=461, compaction requested=true 2023-04-12 15:55:53,653 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4df381dfb8e2792d3aca65faf2673530: 2023-04-12 15:55:53,654 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:55:53,654 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 15:55:53,654 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 15:55:53,654 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:55:53,654 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:55:53,655 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35102 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 15:55:53,655 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35102 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 15:55:53,655 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1912): 4df381dfb8e2792d3aca65faf2673530/A is initiating minor compaction (all files) 2023-04-12 15:55:53,655 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1912): 4df381dfb8e2792d3aca65faf2673530/B is initiating minor compaction (all files) 2023-04-12 15:55:53,655 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4df381dfb8e2792d3aca65faf2673530/A in TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:53,655 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4df381dfb8e2792d3aca65faf2673530/B in TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:53,655 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/99a9ad6cfbab4e659f38d9d9b9b3d270, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/ec600f2c00ff478ba737ed5f2cc7479a, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/c3b12e3a43b245b59c783ca2da5b739e] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp, totalSize=34.3 K 2023-04-12 15:55:53,655 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/61fec2329fdc4306ac0637dd065ea208, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/035b5dfb907c47069b2ea091cf343ff0, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/9daf49a342ae46559cced96796e63846] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp, totalSize=34.3 K 2023-04-12 15:55:53,656 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 99a9ad6cfbab4e659f38d9d9b9b3d270, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=427, earliestPutTs=1681314950885 2023-04-12 15:55:53,656 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 61fec2329fdc4306ac0637dd065ea208, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=427, earliestPutTs=1681314950885 2023-04-12 15:55:53,657 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting ec600f2c00ff478ba737ed5f2cc7479a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=448, earliestPutTs=1681314952043 2023-04-12 15:55:53,658 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 035b5dfb907c47069b2ea091cf343ff0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=448, earliestPutTs=1681314952043 2023-04-12 15:55:53,658 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting c3b12e3a43b245b59c783ca2da5b739e, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=461, earliestPutTs=1681314953193 2023-04-12 15:55:53,658 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 9daf49a342ae46559cced96796e63846, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=461, earliestPutTs=1681314953193 2023-04-12 15:55:53,664 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] throttle.PressureAwareThroughputController(145): 4df381dfb8e2792d3aca65faf2673530#B#compaction#494 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:55:53,664 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4df381dfb8e2792d3aca65faf2673530#A#compaction#493 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:55:53,674 DEBUG [Thread-3125] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:53,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:53,677 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:53,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:53,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:53,680 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:53,680 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:53,680 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:53,680 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:53,681 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:53,681 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:53,683 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,683 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,683 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:53,683 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:53,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:53,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:53,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:53,684 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:53,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:53,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:53,685 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:53,685 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/A/478f15888d08452ea36c03db8db2db59 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/478f15888d08452ea36c03db8db2db59 2023-04-12 15:55:53,685 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:53,685 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:53,685 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:53,685 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:53,685 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:53,686 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:53,686 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:53,686 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:53,686 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. started... 2023-04-12 15:55:53,686 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4df381dfb8e2792d3aca65faf2673530 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-12 15:55:53,686 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=A 2023-04-12 15:55:53,686 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:55:53,686 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=B 2023-04-12 15:55:53,686 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:55:53,686 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=C 2023-04-12 15:55:53,686 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:55:53,687 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/B/078e1577ea1c42c380bc0c224966c6d1 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/078e1577ea1c42c380bc0c224966c6d1 2023-04-12 15:55:53,689 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4df381dfb8e2792d3aca65faf2673530/A of 4df381dfb8e2792d3aca65faf2673530 into 478f15888d08452ea36c03db8db2db59(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:55:53,689 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4df381dfb8e2792d3aca65faf2673530: 2023-04-12 15:55:53,689 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530., storeName=4df381dfb8e2792d3aca65faf2673530/A, priority=13, startTime=1681314953654; duration=0sec 2023-04-12 15:55:53,689 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:55:53,690 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 15:55:53,691 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35068 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 15:55:53,691 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1912): 4df381dfb8e2792d3aca65faf2673530/C is initiating minor compaction (all files) 2023-04-12 15:55:53,691 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4df381dfb8e2792d3aca65faf2673530/C in TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:53,691 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/6e7333c2a545494d8452781dab218e1f, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/d02c7ea22ce34295b782128619720dae, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/06b0d58b1a7b4e6c9e199aeada295bae] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp, totalSize=34.2 K 2023-04-12 15:55:53,691 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 6e7333c2a545494d8452781dab218e1f, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=427, earliestPutTs=1681314950885 2023-04-12 15:55:53,691 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting d02c7ea22ce34295b782128619720dae, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=448, earliestPutTs=1681314952043 2023-04-12 15:55:53,692 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 06b0d58b1a7b4e6c9e199aeada295bae, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=461, earliestPutTs=1681314953193 2023-04-12 15:55:53,694 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4df381dfb8e2792d3aca65faf2673530/B of 4df381dfb8e2792d3aca65faf2673530 into 078e1577ea1c42c380bc0c224966c6d1(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:55:53,694 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4df381dfb8e2792d3aca65faf2673530: 2023-04-12 15:55:53,694 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530., storeName=4df381dfb8e2792d3aca65faf2673530/B, priority=13, startTime=1681314953654; duration=0sec 2023-04-12 15:55:53,694 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:55:53,698 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=497 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/A/482a046ad0b74c6bb026a981aa09f873 2023-04-12 15:55:53,698 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4df381dfb8e2792d3aca65faf2673530#C#compaction#496 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:55:53,719 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=497 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/B/cb82fa2cd7eb4d0ea39b88af5be5e899 2023-04-12 15:55:53,732 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=497 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/C/84e1281e7fc2430c86e3531719541b35 2023-04-12 15:55:53,736 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/A/482a046ad0b74c6bb026a981aa09f873 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/482a046ad0b74c6bb026a981aa09f873 2023-04-12 15:55:53,744 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/482a046ad0b74c6bb026a981aa09f873, entries=150, sequenceid=497, filesize=11.9 K 2023-04-12 15:55:53,745 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/B/cb82fa2cd7eb4d0ea39b88af5be5e899 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/cb82fa2cd7eb4d0ea39b88af5be5e899 2023-04-12 15:55:53,756 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/cb82fa2cd7eb4d0ea39b88af5be5e899, entries=150, sequenceid=497, filesize=11.9 K 2023-04-12 15:55:53,757 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/C/84e1281e7fc2430c86e3531719541b35 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/84e1281e7fc2430c86e3531719541b35 2023-04-12 15:55:53,774 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/84e1281e7fc2430c86e3531719541b35, entries=150, sequenceid=497, filesize=11.9 K 2023-04-12 15:55:53,775 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=0 B/0 for 4df381dfb8e2792d3aca65faf2673530 in 89ms, sequenceid=497, compaction requested=false 2023-04-12 15:55:53,775 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4df381dfb8e2792d3aca65faf2673530: 2023-04-12 15:55:53,775 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:53,775 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:53,775 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:53,775 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,775 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:53,775 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:53,777 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:53,777 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:53,777 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:53,777 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:53,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:53,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:53,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:53,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:53,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:53,782 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:53,782 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21a3dd83[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:53,782 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:53,782 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:53,784 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:53,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:53,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:53,785 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:53,785 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:53,785 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:53,785 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,785 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:53,788 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:53,788 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:53,788 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:53,788 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:53,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:53,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:53,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:53,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:53,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:53,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:53,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:53,791 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:53,791 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:53,791 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:53,791 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:53,791 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:53,791 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:53,793 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:53,793 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:53,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:53,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:53,793 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:53,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:53,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:53,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:53,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:53,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:53,794 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:53,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:53,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:53,795 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:53,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:53,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:53,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:53,800 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:53,801 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:53,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:53,801 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:53,801 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,801 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:53,801 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:53,801 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,801 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:53,801 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:53,801 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:53,801 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:53,801 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:53,801 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:53,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:53,801 DEBUG [Thread-3125] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:53,801 DEBUG [Thread-3125] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:53,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:53,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:53,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on 4df381dfb8e2792d3aca65faf2673530 2023-04-12 15:55:53,856 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4df381dfb8e2792d3aca65faf2673530 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-12 15:55:53,857 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=A 2023-04-12 15:55:53,857 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:53,857 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=B 2023-04-12 15:55:53,857 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:53,857 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=C 2023-04-12 15:55:53,857 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:53,870 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=509 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/A/b81c4df6f5e143c8a042716a465f844f 2023-04-12 15:55:53,881 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=509 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/B/ebde959533554bf2aa71ae822c08909b 2023-04-12 15:55:53,902 DEBUG [Thread-3125] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:53,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:53,904 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:53,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:53,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:53,905 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:53,905 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:53,905 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:53,905 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:53,906 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:53,906 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:53,907 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:53,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:53,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:53,907 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:53,908 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,908 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:53,908 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:53,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:53,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:53,908 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:53,909 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:53,909 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:53,909 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:53,909 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:53,909 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:53,909 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:53,909 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:53,909 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:53,909 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. started... 2023-04-12 15:55:53,909 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. as already flushing 2023-04-12 15:55:53,909 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:53,909 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:53,909 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:53,909 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,909 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:53,909 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:53,911 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:53,912 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:53,912 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:53,912 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681314953900, max timestamp=1681314953910Num uniques 859; 2023-04-12 15:55:53,912 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-12 15:55:53,912 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681314953902, max timestamp=1681314953910Num uniques 859; 2023-04-12 15:55:53,912 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681314953902, max timestamp=1681314953910Num uniques 859; 2023-04-12 15:55:53,912 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-12 15:55:53,913 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,912 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-12 15:55:53,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:53,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:53,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:53,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:53,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:53,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:53,914 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:53,914 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56f4ec04[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:53,914 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:53,915 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:53,917 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:53,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:53,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:53,917 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:53,917 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:53,917 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:53,917 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,918 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:53,919 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,919 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:53,919 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:53,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:53,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:53,919 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:53,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:53,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:53,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:53,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:53,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:53,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:53,921 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:53,922 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:53,922 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:53,922 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:53,922 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:53,922 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:53,924 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:53,924 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:53,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:53,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:53,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:53,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:53,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:53,924 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:53,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:53,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:53,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:53,924 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:53,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:53,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:53,925 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:53,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:53,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:53,930 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,930 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:53,930 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:53,930 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:53,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:53,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:53,930 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:53,930 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,930 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:53,930 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:53,930 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:53,930 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:53,930 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:53,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:53,930 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:53,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:53,930 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:53,931 DEBUG [Thread-3125] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:53,931 DEBUG [Thread-3125] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:53,953 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:53,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55260 deadline: 1681315013944, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,953 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:53,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55306 deadline: 1681315013944, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,965 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:53,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55336 deadline: 1681315013952, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,966 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:53,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681315013953, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:53,966 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:53,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55340 deadline: 1681315013953, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:54,031 DEBUG [Thread-3125] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:54,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:54,033 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:54,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:54,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:54,034 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:54,034 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:54,034 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:54,034 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:54,034 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:54,034 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:54,036 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:54,036 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:54,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:54,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:54,037 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:54,037 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:54,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:54,037 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:54,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:54,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:54,038 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:54,038 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:54,038 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:54,038 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:54,038 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:54,038 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:54,038 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:54,038 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:54,038 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:54,038 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. started... 2023-04-12 15:55:54,038 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. as already flushing 2023-04-12 15:55:54,038 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:54,038 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:54,038 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:54,038 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:54,038 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:54,038 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:54,040 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:54,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:54,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:54,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:54,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:54,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:54,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:54,041 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:54,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:54,041 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:54,041 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:54,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:54,041 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:54,041 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@cd9f2f[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:54,041 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:54,042 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:54,043 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:54,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:54,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:54,043 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:54,043 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:54,043 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:54,043 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:54,043 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:54,044 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:54,044 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:54,044 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:54,044 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:54,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:54,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:54,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:54,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:54,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:54,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:54,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:54,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:54,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:54,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:54,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:54,047 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:54,047 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:54,047 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:54,047 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:54,047 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:54,047 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:54,049 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:54,049 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:54,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:54,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:54,050 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:54,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:54,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:54,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:54,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:54,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:54,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:54,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:54,050 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:54,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:54,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:54,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:54,051 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:54,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:54,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:54,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:54,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:54,054 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:54,054 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:54,054 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:54,054 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:54,054 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:54,054 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:54,054 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:54,054 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:54,054 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:54,054 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:54,054 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:54,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:54,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:54,054 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:54,054 DEBUG [Thread-3125] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:54,054 DEBUG [Thread-3125] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:54,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:54,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:54,063 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:54,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55260 deadline: 1681315014054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:54,063 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:54,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55306 deadline: 1681315014054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:54,076 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:54,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55336 deadline: 1681315014067, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:54,076 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:54,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681315014067, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:54,077 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:54,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55340 deadline: 1681315014067, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:54,125 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/C/aa85961bdecb4c63bf0d35e6d047608a as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/aa85961bdecb4c63bf0d35e6d047608a 2023-04-12 15:55:54,130 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4df381dfb8e2792d3aca65faf2673530/C of 4df381dfb8e2792d3aca65faf2673530 into aa85961bdecb4c63bf0d35e6d047608a(size=12.9 K), total size for store is 24.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:55:54,130 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4df381dfb8e2792d3aca65faf2673530: 2023-04-12 15:55:54,130 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530., storeName=4df381dfb8e2792d3aca65faf2673530/C, priority=13, startTime=1681314953654; duration=0sec 2023-04-12 15:55:54,130 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:55:54,155 DEBUG [Thread-3125] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:54,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:54,156 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:54,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:54,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:54,158 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:54,158 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:54,158 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:54,158 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:54,158 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:54,158 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:54,160 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:54,160 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:54,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:54,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:54,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:54,161 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:54,161 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:54,161 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:54,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:54,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:54,161 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:54,162 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:54,162 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:54,162 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:54,162 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:54,162 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:54,162 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:54,162 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:54,162 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:54,162 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. started... 2023-04-12 15:55:54,162 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. as already flushing 2023-04-12 15:55:54,162 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:54,162 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:54,162 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:54,162 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:54,162 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:54,162 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:54,164 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:54,164 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:54,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:54,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:54,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:54,164 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:54,164 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:54,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:54,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:54,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:54,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:54,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:54,165 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:54,165 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f110a0b[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:54,165 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:54,165 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:54,167 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:54,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:54,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:54,168 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:54,168 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:54,168 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:54,168 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:54,168 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:54,169 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:54,169 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:54,169 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:54,169 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:54,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:54,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:54,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:54,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:54,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:54,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:54,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:54,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:54,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:54,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:54,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:54,172 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:54,172 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:54,172 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:54,172 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:54,172 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:54,172 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:54,173 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:54,173 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:54,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:54,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:54,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:54,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:54,173 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:54,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:54,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:54,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:54,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:54,174 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:54,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:54,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:54,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:54,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:54,175 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:54,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:54,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:54,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:54,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:54,180 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:54,180 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:54,180 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:54,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:54,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:54,180 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:54,180 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:54,181 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:54,181 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:54,181 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:54,181 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:54,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:54,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:54,182 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:54,182 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:54,182 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:54,182 DEBUG [Thread-3125] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:54,182 DEBUG [Thread-3125] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:54,184 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:54,268 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:54,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55260 deadline: 1681315014264, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:54,269 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:54,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55306 deadline: 1681315014265, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:54,282 DEBUG [Thread-3125] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:54,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:54,284 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:54,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55340 deadline: 1681315014278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:54,284 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:54,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55336 deadline: 1681315014278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:54,284 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:54,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681315014279, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:54,286 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:54,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:54,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:54,288 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:54,288 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:54,288 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:54,288 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:54,289 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:54,289 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:54,291 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:54,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:54,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:54,291 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:54,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:54,292 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:54,292 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:54,292 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:54,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:54,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:54,292 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:54,296 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:54,296 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:54,296 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:54,296 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:54,296 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:54,300 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:54,300 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:54,300 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:54,300 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. started... 2023-04-12 15:55:54,301 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. as already flushing 2023-04-12 15:55:54,301 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:54,302 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=509 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/C/3408ae57524c4728bd7330af5cd723f8 2023-04-12 15:55:54,303 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:54,303 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:54,303 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:54,303 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:54,303 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:54,305 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:54,305 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:54,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:54,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:54,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:54,305 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:54,305 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:54,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:54,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:54,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:54,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:54,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:54,308 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:54,309 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6197af[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:54,309 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:54,309 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:54,308 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/A/b81c4df6f5e143c8a042716a465f844f as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/b81c4df6f5e143c8a042716a465f844f 2023-04-12 15:55:54,311 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:54,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:54,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:54,311 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:54,311 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:54,311 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:54,311 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:54,311 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:54,313 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:54,313 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:54,313 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:54,313 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:54,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:54,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:54,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:54,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:54,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:54,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:54,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:54,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:54,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:54,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:54,316 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/b81c4df6f5e143c8a042716a465f844f, entries=250, sequenceid=509, filesize=16.7 K 2023-04-12 15:55:54,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:54,317 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:54,317 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:54,317 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:54,317 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:54,317 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:54,317 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:54,317 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/B/ebde959533554bf2aa71ae822c08909b as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/ebde959533554bf2aa71ae822c08909b 2023-04-12 15:55:54,318 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:54,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:54,319 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:54,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:54,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:54,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:54,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:54,319 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:54,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:54,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:54,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:54,320 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:54,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:54,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:54,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:54,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:54,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:54,334 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:54,334 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/ebde959533554bf2aa71ae822c08909b, entries=150, sequenceid=509, filesize=11.9 K 2023-04-12 15:55:54,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:54,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:54,335 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/C/3408ae57524c4728bd7330af5cd723f8 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/3408ae57524c4728bd7330af5cd723f8 2023-04-12 15:55:54,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:54,337 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:54,337 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:54,337 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:54,337 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:54,337 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:54,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:54,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:54,337 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:54,338 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:54,338 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:54,338 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:54,338 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:54,337 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:54,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:54,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:54,337 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:54,340 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:54,340 DEBUG [Thread-3125] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:54,340 DEBUG [Thread-3125] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:54,342 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/3408ae57524c4728bd7330af5cd723f8, entries=150, sequenceid=509, filesize=11.9 K 2023-04-12 15:55:54,343 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=136.42 KB/139690 for 4df381dfb8e2792d3aca65faf2673530 in 487ms, sequenceid=509, compaction requested=true 2023-04-12 15:55:54,343 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4df381dfb8e2792d3aca65faf2673530: 2023-04-12 15:55:54,343 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:55:54,343 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 15:55:54,343 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:55:54,343 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 15:55:54,343 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:55:54,345 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 42528 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 15:55:54,345 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37648 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 15:55:54,345 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1912): 4df381dfb8e2792d3aca65faf2673530/A is initiating minor compaction (all files) 2023-04-12 15:55:54,345 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1912): 4df381dfb8e2792d3aca65faf2673530/B is initiating minor compaction (all files) 2023-04-12 15:55:54,345 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4df381dfb8e2792d3aca65faf2673530/A in TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:54,345 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4df381dfb8e2792d3aca65faf2673530/B in TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:54,345 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/478f15888d08452ea36c03db8db2db59, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/482a046ad0b74c6bb026a981aa09f873, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/b81c4df6f5e143c8a042716a465f844f] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp, totalSize=41.5 K 2023-04-12 15:55:54,345 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/078e1577ea1c42c380bc0c224966c6d1, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/cb82fa2cd7eb4d0ea39b88af5be5e899, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/ebde959533554bf2aa71ae822c08909b] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp, totalSize=36.8 K 2023-04-12 15:55:54,345 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 478f15888d08452ea36c03db8db2db59, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=461, earliestPutTs=1681314952043 2023-04-12 15:55:54,346 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 078e1577ea1c42c380bc0c224966c6d1, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=461, earliestPutTs=1681314952043 2023-04-12 15:55:54,346 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 482a046ad0b74c6bb026a981aa09f873, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=497, earliestPutTs=1681314953268 2023-04-12 15:55:54,346 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting cb82fa2cd7eb4d0ea39b88af5be5e899, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=497, earliestPutTs=1681314953268 2023-04-12 15:55:54,346 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting b81c4df6f5e143c8a042716a465f844f, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=509, earliestPutTs=1681314953844 2023-04-12 15:55:54,346 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting ebde959533554bf2aa71ae822c08909b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=509, earliestPutTs=1681314953851 2023-04-12 15:55:54,351 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] throttle.PressureAwareThroughputController(145): 4df381dfb8e2792d3aca65faf2673530#B#compaction#503 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:55:54,352 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4df381dfb8e2792d3aca65faf2673530#A#compaction#502 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:55:54,388 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/A/db64391f2fee420e916bff09b38f6d40 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/db64391f2fee420e916bff09b38f6d40 2023-04-12 15:55:54,394 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4df381dfb8e2792d3aca65faf2673530/A of 4df381dfb8e2792d3aca65faf2673530 into db64391f2fee420e916bff09b38f6d40(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:55:54,394 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4df381dfb8e2792d3aca65faf2673530: 2023-04-12 15:55:54,394 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530., storeName=4df381dfb8e2792d3aca65faf2673530/A, priority=13, startTime=1681314954343; duration=0sec 2023-04-12 15:55:54,394 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:55:54,395 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 15:55:54,395 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37614 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 15:55:54,396 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1912): 4df381dfb8e2792d3aca65faf2673530/C is initiating minor compaction (all files) 2023-04-12 15:55:54,396 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4df381dfb8e2792d3aca65faf2673530/C in TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:54,396 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/aa85961bdecb4c63bf0d35e6d047608a, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/84e1281e7fc2430c86e3531719541b35, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/3408ae57524c4728bd7330af5cd723f8] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp, totalSize=36.7 K 2023-04-12 15:55:54,396 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting aa85961bdecb4c63bf0d35e6d047608a, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=461, earliestPutTs=1681314952043 2023-04-12 15:55:54,397 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 84e1281e7fc2430c86e3531719541b35, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=497, earliestPutTs=1681314953268 2023-04-12 15:55:54,397 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 3408ae57524c4728bd7330af5cd723f8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=509, earliestPutTs=1681314953851 2023-04-12 15:55:54,405 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4df381dfb8e2792d3aca65faf2673530#C#compaction#504 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:55:54,421 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/C/80821659e1dd46e5991805b1a0bd4220 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/80821659e1dd46e5991805b1a0bd4220 2023-04-12 15:55:54,427 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4df381dfb8e2792d3aca65faf2673530/C of 4df381dfb8e2792d3aca65faf2673530 into 80821659e1dd46e5991805b1a0bd4220(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:55:54,427 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4df381dfb8e2792d3aca65faf2673530: 2023-04-12 15:55:54,427 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530., storeName=4df381dfb8e2792d3aca65faf2673530/C, priority=13, startTime=1681314954343; duration=0sec 2023-04-12 15:55:54,427 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:55:54,440 DEBUG [Thread-3125] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:54,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:54,442 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:54,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:54,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:54,443 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:54,443 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:54,443 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:54,443 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:54,444 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:54,444 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:54,445 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:54,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:54,445 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:54,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:54,446 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:54,446 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:54,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:54,446 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:54,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:54,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:54,446 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:54,448 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:54,448 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:54,448 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:54,448 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:54,448 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:54,448 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:54,448 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:54,448 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:54,448 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. started... 2023-04-12 15:55:54,448 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4df381dfb8e2792d3aca65faf2673530 3/3 column families, dataSize=136.42 KB heapSize=390.06 KB 2023-04-12 15:55:54,449 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=A 2023-04-12 15:55:54,449 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:55:54,449 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=B 2023-04-12 15:55:54,449 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:55:54,449 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=C 2023-04-12 15:55:54,449 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:55:54,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on 4df381dfb8e2792d3aca65faf2673530 2023-04-12 15:55:54,577 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. as already flushing 2023-04-12 15:55:54,594 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:54,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55340 deadline: 1681315014588, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:54,595 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:54,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55306 deadline: 1681315014593, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:54,600 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:54,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55260 deadline: 1681315014594, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:54,601 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:54,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55336 deadline: 1681315014594, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:54,601 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:54,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681315014595, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:54,698 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:54,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55306 deadline: 1681315014696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:54,709 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:54,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55260 deadline: 1681315014701, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:54,709 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:54,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55336 deadline: 1681315014702, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:54,709 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:54,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681315014702, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:54,778 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/B/041d95a3ca8743dfab3e58fe626137f4 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/041d95a3ca8743dfab3e58fe626137f4 2023-04-12 15:55:54,782 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4df381dfb8e2792d3aca65faf2673530/B of 4df381dfb8e2792d3aca65faf2673530 into 041d95a3ca8743dfab3e58fe626137f4(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:55:54,782 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4df381dfb8e2792d3aca65faf2673530: 2023-04-12 15:55:54,782 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530., storeName=4df381dfb8e2792d3aca65faf2673530/B, priority=13, startTime=1681314954343; duration=0sec 2023-04-12 15:55:54,782 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:55:54,859 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=549 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/A/47d1715099b246619a48639dd7d3fd8b 2023-04-12 15:55:54,903 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:54,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55306 deadline: 1681315014900, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:54,915 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:54,915 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:54,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681315014910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:54,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55260 deadline: 1681315014911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:54,916 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:54,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55336 deadline: 1681315014911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:55,100 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:55,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55340 deadline: 1681315015097, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:55,206 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:55,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55306 deadline: 1681315015205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:55,220 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:55,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681315015216, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:55,221 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:55,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55336 deadline: 1681315015217, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:55,221 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:55,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55260 deadline: 1681315015218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:55,273 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=549 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/B/fa12f6a5c0b74a30bcf6879275e79c91 2023-04-12 15:55:55,684 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=549 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/C/f6104a9563bf4827bfa9bbaadd207566 2023-04-12 15:55:55,689 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/A/47d1715099b246619a48639dd7d3fd8b as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/47d1715099b246619a48639dd7d3fd8b 2023-04-12 15:55:55,692 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/47d1715099b246619a48639dd7d3fd8b, entries=150, sequenceid=549, filesize=11.9 K 2023-04-12 15:55:55,693 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/B/fa12f6a5c0b74a30bcf6879275e79c91 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/fa12f6a5c0b74a30bcf6879275e79c91 2023-04-12 15:55:55,697 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/fa12f6a5c0b74a30bcf6879275e79c91, entries=150, sequenceid=549, filesize=11.9 K 2023-04-12 15:55:55,698 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/C/f6104a9563bf4827bfa9bbaadd207566 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/f6104a9563bf4827bfa9bbaadd207566 2023-04-12 15:55:55,702 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/f6104a9563bf4827bfa9bbaadd207566, entries=150, sequenceid=549, filesize=11.9 K 2023-04-12 15:55:55,703 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~136.42 KB/139690, heapSize ~390.02 KB/399376, currentSize=53.67 KB/54960 for 4df381dfb8e2792d3aca65faf2673530 in 1254ms, sequenceid=549, compaction requested=false 2023-04-12 15:55:55,703 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4df381dfb8e2792d3aca65faf2673530: 2023-04-12 15:55:55,703 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:55,703 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:55,703 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:55,703 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:55,703 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:55,703 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:55,705 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:55,705 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:55,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:55,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:55,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:55,705 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:55,705 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:55,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:55,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:55,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:55,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:55,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:55,707 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:55,707 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7184df6b[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:55,707 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:55,707 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:55,708 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:55,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:55,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:55,709 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:55,709 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:55,709 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:55,709 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:55,709 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:55,711 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:55,711 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:55,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:55,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:55,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:55,711 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:55,711 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:55,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:55,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:55,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:55,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:55,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:55,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:55,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:55,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:55,714 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:55,714 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:55,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on 4df381dfb8e2792d3aca65faf2673530 2023-04-12 15:55:55,714 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:55,714 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:55,714 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:55,714 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:55,714 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4df381dfb8e2792d3aca65faf2673530 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-12 15:55:55,714 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=A 2023-04-12 15:55:55,714 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:55,715 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=B 2023-04-12 15:55:55,715 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:55,715 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=C 2023-04-12 15:55:55,715 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:55,718 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:55,718 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:55,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:55,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:55,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:55,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:55,719 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:55,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:55,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:55,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:55,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:55,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:55,720 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:55,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:55,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:55,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:55,722 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:55,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:55,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:55,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:55,725 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:55,725 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:55,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:55,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:55,725 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:55,725 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:55,725 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:55,725 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:55,725 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:55,725 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:55,725 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:55,725 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:55,725 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:55,725 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:55,725 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:55,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:55,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:55,725 DEBUG [Thread-3125] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:55,725 DEBUG [Thread-3125] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:55,761 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:55,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55336 deadline: 1681315015751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:55,762 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:55,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55306 deadline: 1681315015753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:55,762 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:55,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681315015758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:55,767 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:55,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55260 deadline: 1681315015759, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:55,825 DEBUG [Thread-3125] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:55,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:55,827 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:55,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:55,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:55,829 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:55,829 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:55,829 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:55,829 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:55,829 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:55,829 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:55,831 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:55,831 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:55,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:55,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:55,831 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:55,831 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:55,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:55,831 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:55,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:55,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:55,832 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:55,832 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:55,832 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:55,832 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:55,832 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:55,832 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:55,832 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:55,832 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:55,832 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:55,833 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. started... 2023-04-12 15:55:55,833 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. as already flushing 2023-04-12 15:55:55,833 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:55,833 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:55,833 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:55,833 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:55,833 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:55,833 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:55,835 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:55,835 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:55,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:55,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:55,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:55,835 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:55,835 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:55,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:55,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:55,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:55,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:55,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:55,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:55,838 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:55,838 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:55,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d5c1293[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:55,842 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:55,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:55,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:55,842 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:55,842 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:55,842 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:55,842 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:55,842 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:55,844 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:55,844 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:55,844 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:55,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:55,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:55,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:55,844 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:55,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:55,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:55,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:55,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:55,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:55,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:55,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:55,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:55,848 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:55,848 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:55,848 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:55,848 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:55,848 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:55,848 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:55,849 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:55,849 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:55,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:55,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:55,849 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:55,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:55,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:55,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:55,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:55,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:55,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:55,850 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:55,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:55,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:55,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:55,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:55,851 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:55,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:55,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:55,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:55,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:55,853 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:55,853 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:55,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:55,853 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:55,853 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:55,853 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:55,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:55,853 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:55,853 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:55,855 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:55,855 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:55,855 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:55,855 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:55,855 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:55,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:55,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:55,856 DEBUG [Thread-3125] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:55,856 DEBUG [Thread-3125] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:55,856 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:55,868 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:55,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55336 deadline: 1681315015863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:55,868 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:55,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55306 deadline: 1681315015863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:55,869 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:55,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681315015863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:55,874 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:55,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55260 deadline: 1681315015868, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:55,956 DEBUG [Thread-3125] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:55,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:55,958 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:55,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:55,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:55,960 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:55,960 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:55,960 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:55,960 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:55,960 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:55,960 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:55,965 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:55,965 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:55,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:55,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:55,965 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:55,965 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:55,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:55,966 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:55,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:55,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:55,966 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:55,966 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:55,966 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:55,966 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:55,967 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:55,967 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:55,967 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:55,967 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:55,967 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:55,967 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. started... 2023-04-12 15:55:55,967 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. as already flushing 2023-04-12 15:55:55,967 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:55,967 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:55,967 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:55,967 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:55,967 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:55,967 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:55,969 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:55,969 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:55,969 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:55,972 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:55,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:55,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:55,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:55,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:55,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:55,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:55,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:55,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:55,973 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:55,973 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a2a0261[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:55,973 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:55,973 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:55,975 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:55,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:55,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:55,975 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:55,975 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:55,975 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:55,975 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:55,975 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:55,976 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:55,976 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:55,976 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:55,976 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:55,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:55,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:55,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:55,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:55,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:55,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:55,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:55,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:55,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:55,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:55,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:55,981 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:55,981 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:55,981 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:55,981 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:55,981 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:55,981 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:55,983 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:55,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:55,983 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:55,984 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:55,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:55,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:55,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:55,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:55,984 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:55,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:55,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:55,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:55,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:55,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:55,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:55,985 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:55,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:55,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:55,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:55,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:55,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:55,987 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:55,987 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:55,987 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:55,987 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:55,988 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:55,987 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:55,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:55,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:55,988 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:55,988 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:55,988 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:55,988 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:55,988 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:55,988 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:55,988 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:55,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:55,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:55,988 DEBUG [Thread-3125] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:55,988 DEBUG [Thread-3125] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:56,077 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:56,077 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:56,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55336 deadline: 1681315016069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:56,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55306 deadline: 1681315016069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:56,077 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:56,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681315016071, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:56,080 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:56,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55260 deadline: 1681315016076, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:56,088 DEBUG [Thread-3125] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:56,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:56,091 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:56,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:56,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:56,092 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:56,092 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:56,092 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:56,092 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:56,092 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:56,092 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:56,094 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:56,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:56,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:56,094 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:56,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:56,095 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:56,095 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:56,095 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:56,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:56,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:56,096 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:56,096 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:56,096 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:56,096 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:56,096 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:56,096 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:56,097 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:56,097 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:56,097 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:56,097 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. started... 2023-04-12 15:55:56,097 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. as already flushing 2023-04-12 15:55:56,097 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:56,097 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:56,097 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:56,097 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:56,097 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:56,098 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:56,099 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:56,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:56,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:56,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:56,099 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:56,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:56,100 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:56,100 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:56,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:56,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:56,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:56,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:56,102 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:56,102 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@267a3464[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:56,102 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:56,103 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:56,106 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:56,106 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:56,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55340 deadline: 1681315016102, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:56,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:56,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:56,106 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:56,106 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:56,107 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:56,107 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:56,107 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:56,109 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:56,109 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:56,109 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:56,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:56,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:56,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:56,109 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:56,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:56,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:56,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:56,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:56,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:56,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:56,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:56,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:56,113 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:56,113 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:56,113 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:56,113 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:56,113 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:56,113 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:56,114 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:56,114 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:56,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:56,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:56,114 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:56,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:56,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:56,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:56,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:56,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:56,115 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:56,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:56,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:56,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:56,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:56,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:56,118 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:56,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:56,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:56,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:56,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:56,120 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:56,120 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:56,120 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:56,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:56,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:56,120 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:56,120 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:56,120 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:56,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:56,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:56,120 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:56,121 DEBUG [Thread-3125] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:56,121 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:56,121 DEBUG [Thread-3125] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:56,120 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:56,121 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:56,121 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:56,121 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:56,121 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:56,134 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=562 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/A/90db606fc30447f2bac31ff8cad3541c 2023-04-12 15:55:56,221 DEBUG [Thread-3125] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:56,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:56,223 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:56,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:56,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:56,224 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:56,224 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:56,224 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:56,224 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:56,224 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:56,225 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:56,227 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:56,227 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:56,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:56,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:56,227 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:56,227 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:56,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:56,228 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:56,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:56,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:56,228 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:56,228 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:56,228 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:56,228 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:56,229 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:56,229 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:56,230 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:56,230 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:56,230 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:56,230 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. started... 2023-04-12 15:55:56,230 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. as already flushing 2023-04-12 15:55:56,230 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:56,230 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:56,230 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:56,230 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:56,230 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:56,230 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:56,232 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:56,232 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:56,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:56,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:56,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:56,232 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:56,232 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:56,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:56,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:56,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:56,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:56,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:56,234 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:56,234 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71d2d847[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:56,234 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:56,234 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:56,235 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:56,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:56,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:56,235 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:56,235 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:56,235 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:56,235 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:56,235 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:56,236 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:56,236 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:56,236 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:56,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:56,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:56,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:56,236 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:56,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:56,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:56,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:56,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:56,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:56,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:56,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:56,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:56,239 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:56,239 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:56,239 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:56,239 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:56,239 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:56,239 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:56,241 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:56,241 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:56,241 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:56,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:56,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:56,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:56,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:56,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:56,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:56,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:56,242 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:56,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:56,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:56,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:56,243 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:56,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:56,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:56,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:56,245 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:56,246 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:56,246 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:56,246 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:56,246 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:56,246 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:56,246 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:56,246 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:56,246 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:56,246 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:56,246 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:56,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:56,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:56,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:56,246 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:56,246 DEBUG [Thread-3125] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:56,246 DEBUG [Thread-3125] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:56,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:56,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:56,247 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:56,346 DEBUG [Thread-3125] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:56,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:56,348 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:56,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:56,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:56,350 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:56,350 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:56,350 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:56,350 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:56,350 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:56,350 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:56,352 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:56,352 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:56,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:56,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:56,352 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:56,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:56,352 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:56,352 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:56,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:56,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:56,353 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:56,353 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:56,353 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:56,353 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:56,353 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:56,353 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:56,353 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:56,353 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:56,354 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:56,354 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. started... 2023-04-12 15:55:56,354 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. as already flushing 2023-04-12 15:55:56,354 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:56,354 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:56,354 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:56,354 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:56,354 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:56,354 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:56,356 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:56,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:56,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:56,356 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:56,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:56,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:56,357 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:56,357 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:56,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:56,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:56,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:56,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:56,358 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:56,358 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ce79464[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:56,358 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:56,359 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:56,369 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:56,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:56,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:56,370 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:56,370 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:56,370 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:56,370 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:56,370 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:56,371 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:56,371 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:56,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:56,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:56,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:56,371 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:56,371 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:56,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:56,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:56,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:56,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:56,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:56,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:56,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:56,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:56,376 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:56,376 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:56,376 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:56,376 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:56,376 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:56,376 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:56,378 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:56,378 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:56,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:56,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:56,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:56,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:56,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:56,378 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:56,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:56,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:56,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:56,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:56,379 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:56,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:56,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:56,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:56,381 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:56,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:56,381 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:56,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55306 deadline: 1681315016378, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:56,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:56,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:56,383 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:56,383 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:56,383 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:56,383 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:56,383 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:56,383 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:56,383 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:56,383 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:56,383 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:56,383 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:56,383 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:56,383 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:56,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:56,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:56,384 DEBUG [Thread-3125] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:56,384 DEBUG [Thread-3125] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:56,384 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:56,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:56,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:56,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55336 deadline: 1681315016379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:56,384 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:56,384 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:56,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681315016380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:56,386 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:56,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55260 deadline: 1681315016382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:56,484 DEBUG [Thread-3125] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:56,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:56,486 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:56,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:56,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:56,487 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:56,487 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:56,487 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:56,487 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:56,487 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:56,487 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:56,490 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:56,490 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:56,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:56,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:56,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:56,491 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:56,491 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:56,491 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:56,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:56,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:56,491 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:56,492 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:56,492 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:56,492 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:56,492 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:56,492 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:56,492 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:56,492 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:56,492 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:56,492 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. started... 2023-04-12 15:55:56,492 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. as already flushing 2023-04-12 15:55:56,492 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:56,492 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:56,492 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:56,492 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:56,492 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:56,493 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:56,494 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:56,494 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:56,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:56,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:56,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:56,495 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:56,495 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:56,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:56,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:56,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:56,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:56,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:56,496 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:56,496 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76a4b693[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:56,496 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:56,496 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:56,497 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:56,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:56,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:56,497 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:56,498 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:56,498 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:56,498 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:56,498 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:56,499 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:56,499 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:56,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:56,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:56,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:56,499 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:56,499 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:56,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:56,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:56,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:56,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:56,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:56,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:56,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:56,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:56,501 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:56,501 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:56,502 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:56,502 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:56,502 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:56,502 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:56,504 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:56,504 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:56,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:56,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:56,504 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:56,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:56,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:56,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:56,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:56,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:56,505 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:56,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:56,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:56,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:56,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:56,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:56,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:56,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:56,508 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:56,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:56,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:56,511 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:56,511 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:56,511 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:56,511 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:56,511 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:56,511 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:56,511 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:56,511 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:56,511 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:56,511 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:56,511 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:56,511 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:56,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:56,512 DEBUG [Thread-3125] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:56,512 DEBUG [Thread-3125] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:56,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:56,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:56,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:56,544 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=562 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/B/2f6cffef02bd4ea7b8059df0fef34c3b 2023-04-12 15:55:56,555 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=562 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/C/7c31f63dc8ec488aa81f08ba1d15feb2 2023-04-12 15:55:56,559 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/A/90db606fc30447f2bac31ff8cad3541c as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/90db606fc30447f2bac31ff8cad3541c 2023-04-12 15:55:56,565 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/90db606fc30447f2bac31ff8cad3541c, entries=200, sequenceid=562, filesize=14.3 K 2023-04-12 15:55:56,566 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/B/2f6cffef02bd4ea7b8059df0fef34c3b as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/2f6cffef02bd4ea7b8059df0fef34c3b 2023-04-12 15:55:56,569 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/2f6cffef02bd4ea7b8059df0fef34c3b, entries=150, sequenceid=562, filesize=11.9 K 2023-04-12 15:55:56,570 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/C/7c31f63dc8ec488aa81f08ba1d15feb2 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/7c31f63dc8ec488aa81f08ba1d15feb2 2023-04-12 15:55:56,573 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/7c31f63dc8ec488aa81f08ba1d15feb2, entries=150, sequenceid=562, filesize=11.9 K 2023-04-12 15:55:56,574 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 4df381dfb8e2792d3aca65faf2673530 in 860ms, sequenceid=562, compaction requested=true 2023-04-12 15:55:56,574 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4df381dfb8e2792d3aca65faf2673530: 2023-04-12 15:55:56,574 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:55:56,574 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 15:55:56,574 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 15:55:56,574 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:55:56,575 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:55:56,575 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40190 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 15:55:56,576 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1912): 4df381dfb8e2792d3aca65faf2673530/A is initiating minor compaction (all files) 2023-04-12 15:55:56,576 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4df381dfb8e2792d3aca65faf2673530/A in TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:56,576 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/db64391f2fee420e916bff09b38f6d40, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/47d1715099b246619a48639dd7d3fd8b, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/90db606fc30447f2bac31ff8cad3541c] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp, totalSize=39.2 K 2023-04-12 15:55:56,576 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37750 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 15:55:56,576 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1912): 4df381dfb8e2792d3aca65faf2673530/B is initiating minor compaction (all files) 2023-04-12 15:55:56,576 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4df381dfb8e2792d3aca65faf2673530/B in TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:56,576 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/041d95a3ca8743dfab3e58fe626137f4, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/fa12f6a5c0b74a30bcf6879275e79c91, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/2f6cffef02bd4ea7b8059df0fef34c3b] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp, totalSize=36.9 K 2023-04-12 15:55:56,576 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting db64391f2fee420e916bff09b38f6d40, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=509, earliestPutTs=1681314953851 2023-04-12 15:55:56,576 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 041d95a3ca8743dfab3e58fe626137f4, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=509, earliestPutTs=1681314953851 2023-04-12 15:55:56,576 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 47d1715099b246619a48639dd7d3fd8b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=549, earliestPutTs=1681314953943 2023-04-12 15:55:56,577 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting fa12f6a5c0b74a30bcf6879275e79c91, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=549, earliestPutTs=1681314953943 2023-04-12 15:55:56,577 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 90db606fc30447f2bac31ff8cad3541c, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=562, earliestPutTs=1681314954576 2023-04-12 15:55:56,577 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 2f6cffef02bd4ea7b8059df0fef34c3b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=562, earliestPutTs=1681314954593 2023-04-12 15:55:56,583 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4df381dfb8e2792d3aca65faf2673530#A#compaction#511 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:55:56,584 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] throttle.PressureAwareThroughputController(145): 4df381dfb8e2792d3aca65faf2673530#B#compaction#512 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:55:56,600 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/A/f02f5fcee75749cc88ccdece827a3005 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/f02f5fcee75749cc88ccdece827a3005 2023-04-12 15:55:56,604 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4df381dfb8e2792d3aca65faf2673530/A of 4df381dfb8e2792d3aca65faf2673530 into f02f5fcee75749cc88ccdece827a3005(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:55:56,604 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4df381dfb8e2792d3aca65faf2673530: 2023-04-12 15:55:56,604 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530., storeName=4df381dfb8e2792d3aca65faf2673530/A, priority=13, startTime=1681314956574; duration=0sec 2023-04-12 15:55:56,604 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:55:56,604 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 15:55:56,605 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37716 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 15:55:56,605 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1912): 4df381dfb8e2792d3aca65faf2673530/C is initiating minor compaction (all files) 2023-04-12 15:55:56,605 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4df381dfb8e2792d3aca65faf2673530/C in TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:56,605 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/80821659e1dd46e5991805b1a0bd4220, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/f6104a9563bf4827bfa9bbaadd207566, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/7c31f63dc8ec488aa81f08ba1d15feb2] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp, totalSize=36.8 K 2023-04-12 15:55:56,605 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 80821659e1dd46e5991805b1a0bd4220, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=509, earliestPutTs=1681314953851 2023-04-12 15:55:56,606 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting f6104a9563bf4827bfa9bbaadd207566, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=549, earliestPutTs=1681314953943 2023-04-12 15:55:56,606 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 7c31f63dc8ec488aa81f08ba1d15feb2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=562, earliestPutTs=1681314954593 2023-04-12 15:55:56,612 DEBUG [Thread-3125] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:56,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:56,615 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:56,616 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/B/44b25aaec44847d8a77f7c0214e88fea as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/44b25aaec44847d8a77f7c0214e88fea 2023-04-12 15:55:56,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:56,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:56,616 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:56,616 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:56,616 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:56,617 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:56,617 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:56,617 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:56,618 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:56,618 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:56,619 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:56,619 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:56,620 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4df381dfb8e2792d3aca65faf2673530#C#compaction#513 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:55:56,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:56,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:56,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:56,621 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:56,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:56,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:56,622 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:56,624 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4df381dfb8e2792d3aca65faf2673530/B of 4df381dfb8e2792d3aca65faf2673530 into 44b25aaec44847d8a77f7c0214e88fea(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:55:56,624 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4df381dfb8e2792d3aca65faf2673530: 2023-04-12 15:55:56,624 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530., storeName=4df381dfb8e2792d3aca65faf2673530/B, priority=13, startTime=1681314956574; duration=0sec 2023-04-12 15:55:56,624 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:55:56,628 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:56,628 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:56,628 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:56,628 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:56,628 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:56,632 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:56,632 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:56,632 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:56,632 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. started... 2023-04-12 15:55:56,632 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4df381dfb8e2792d3aca65faf2673530 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-12 15:55:56,632 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=A 2023-04-12 15:55:56,633 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:56,633 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=B 2023-04-12 15:55:56,633 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:56,633 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=C 2023-04-12 15:55:56,633 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:56,654 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=584 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/A/5b4f058ea912496f93ca28483cd512a3 2023-04-12 15:55:56,673 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=584 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/B/fe4dda7a758c42f8a1e79edbed7a0a00 2023-04-12 15:55:56,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on 4df381dfb8e2792d3aca65faf2673530 2023-04-12 15:55:56,888 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. as already flushing 2023-04-12 15:55:56,900 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:56,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55336 deadline: 1681315016896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:56,905 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:56,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55306 deadline: 1681315016897, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:56,905 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:56,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55260 deadline: 1681315016897, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:56,909 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:56,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681315016900, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,005 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:57,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55336 deadline: 1681315017001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,009 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:57,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55306 deadline: 1681315017006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,009 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:57,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55260 deadline: 1681315017006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,014 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:57,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681315017010, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,044 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/C/a491231d287c492e888737e4602c64e9 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/a491231d287c492e888737e4602c64e9 2023-04-12 15:55:57,048 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4df381dfb8e2792d3aca65faf2673530/C of 4df381dfb8e2792d3aca65faf2673530 into a491231d287c492e888737e4602c64e9(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:55:57,048 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4df381dfb8e2792d3aca65faf2673530: 2023-04-12 15:55:57,048 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530., storeName=4df381dfb8e2792d3aca65faf2673530/C, priority=13, startTime=1681314956575; duration=0sec 2023-04-12 15:55:57,048 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:55:57,087 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=584 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/C/adc2d7bba1154a52889909aa0a14cca5 2023-04-12 15:55:57,091 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/A/5b4f058ea912496f93ca28483cd512a3 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/5b4f058ea912496f93ca28483cd512a3 2023-04-12 15:55:57,094 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/5b4f058ea912496f93ca28483cd512a3, entries=150, sequenceid=584, filesize=11.9 K 2023-04-12 15:55:57,095 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/B/fe4dda7a758c42f8a1e79edbed7a0a00 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/fe4dda7a758c42f8a1e79edbed7a0a00 2023-04-12 15:55:57,098 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/fe4dda7a758c42f8a1e79edbed7a0a00, entries=150, sequenceid=584, filesize=11.9 K 2023-04-12 15:55:57,099 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/C/adc2d7bba1154a52889909aa0a14cca5 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/adc2d7bba1154a52889909aa0a14cca5 2023-04-12 15:55:57,102 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/adc2d7bba1154a52889909aa0a14cca5, entries=150, sequenceid=584, filesize=11.9 K 2023-04-12 15:55:57,103 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 4df381dfb8e2792d3aca65faf2673530 in 471ms, sequenceid=584, compaction requested=false 2023-04-12 15:55:57,103 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4df381dfb8e2792d3aca65faf2673530: 2023-04-12 15:55:57,103 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:57,103 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:57,103 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:57,103 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,103 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:57,103 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:57,105 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:57,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:57,106 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:57,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:57,106 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:57,106 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:57,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:57,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:57,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:57,108 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:57,108 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10a6537[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:57,108 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:57,108 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:57,110 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:57,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:57,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:57,110 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:57,110 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:57,110 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:57,110 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,111 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:57,112 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:57,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:57,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:57,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:57,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:57,113 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:57,113 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:57,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,113 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:57,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:57,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:57,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:57,115 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:57,115 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:57,115 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:57,115 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:57,115 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:57,115 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:57,118 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:57,118 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:57,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:57,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:57,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:57,118 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:57,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:57,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:57,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:57,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:57,119 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:57,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:57,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:57,120 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:57,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:57,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:57,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:57,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,122 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:57,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:57,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:57,124 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,124 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:57,124 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:57,124 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,124 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:57,124 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:57,124 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:57,124 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:57,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:57,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:57,127 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:57,127 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:57,127 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:57,127 DEBUG [Thread-3125] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:57,127 DEBUG [Thread-3125] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:57,128 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:57,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on 4df381dfb8e2792d3aca65faf2673530 2023-04-12 15:55:57,210 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4df381dfb8e2792d3aca65faf2673530 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-12 15:55:57,210 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=A 2023-04-12 15:55:57,210 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:57,210 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=B 2023-04-12 15:55:57,210 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:57,210 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=C 2023-04-12 15:55:57,210 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:57,228 DEBUG [Thread-3125] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:57,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:57,230 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:57,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:57,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:57,231 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:57,231 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:57,231 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:57,231 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:57,231 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:57,231 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:57,233 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:57,233 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:57,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:57,233 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,233 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:57,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:57,234 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:57,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:57,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:57,234 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:57,234 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:57,234 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:57,234 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:57,235 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:57,235 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:57,235 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:57,235 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:57,235 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:57,235 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. started... 2023-04-12 15:55:57,235 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. as already flushing 2023-04-12 15:55:57,235 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:57,235 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:57,235 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:57,235 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,235 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:57,235 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:57,236 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,237 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:57,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:57,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:57,238 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:57,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:57,238 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:57,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:57,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:57,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:57,239 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:57,239 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66fd85b1[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:57,239 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:57,239 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:57,241 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:57,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:57,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:57,241 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:57,241 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:57,241 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:57,241 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,241 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:57,250 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:57,250 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:57,250 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,250 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:57,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:57,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:57,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:57,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:57,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:57,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:57,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:57,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:57,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:57,252 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:57,252 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:57,252 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:57,252 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:57,252 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:57,254 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:57,254 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:57,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:57,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:57,254 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:57,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:57,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:57,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:57,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:57,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:57,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:57,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:57,255 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:57,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:57,257 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:57,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55260 deadline: 1681315017250, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,257 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:57,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55306 deadline: 1681315017250, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,257 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:57,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:57,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55336 deadline: 1681315017250, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,257 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:57,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681315017251, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,257 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:57,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:57,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,260 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,260 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:57,260 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:57,260 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:57,260 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:57,260 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:57,260 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:57,260 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,260 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:57,260 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:57,260 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:57,260 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:57,260 DEBUG [Thread-3125] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:57,260 DEBUG [Thread-3125] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:57,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:57,260 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:57,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:57,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:57,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:57,361 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:57,361 DEBUG [Thread-3125] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:57,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55260 deadline: 1681315017358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,361 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:57,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55306 deadline: 1681315017358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,361 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:57,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55336 deadline: 1681315017358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:57,362 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:57,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681315017358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,363 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:57,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:57,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:57,364 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:57,364 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:57,364 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:57,364 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:57,365 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:57,365 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:57,366 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:57,366 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:57,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:57,366 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,366 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:57,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:57,367 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:57,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:57,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:57,367 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:57,367 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:57,367 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:57,367 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:57,367 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:57,367 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:57,368 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:57,368 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:57,368 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:57,368 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. started... 2023-04-12 15:55:57,368 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. as already flushing 2023-04-12 15:55:57,368 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:57,368 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:57,368 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:57,368 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,368 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:57,368 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:57,370 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:57,370 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:57,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:57,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:57,371 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:57,371 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:57,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:57,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:57,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:57,373 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:57,373 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@560979be[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:57,373 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:57,374 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:57,374 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:57,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:57,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:57,375 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:57,375 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:57,375 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:57,375 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,375 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:57,376 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,376 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:57,376 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:57,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:57,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:57,376 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:57,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:57,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:57,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:57,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:57,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:57,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:57,378 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:57,378 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:57,378 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:57,378 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:57,379 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:57,379 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:57,380 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:57,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:57,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:57,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:57,380 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:57,380 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:57,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:57,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:57,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:57,381 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:57,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:57,382 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:57,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:57,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:57,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:57,385 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,385 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:57,385 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:57,385 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:57,385 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:57,385 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:57,385 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:57,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:57,385 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:57,386 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:57,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:57,385 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:57,386 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:57,386 DEBUG [Thread-3125] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:57,386 DEBUG [Thread-3125] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:57,386 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:57,386 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:57,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:57,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:57,486 DEBUG [Thread-3125] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:57,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:57,488 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:57,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:57,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:57,489 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:57,489 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:57,489 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:57,489 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:57,490 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:57,490 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:57,491 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:57,491 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:57,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:57,491 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,492 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:57,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:57,492 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:57,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:57,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:57,492 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:57,493 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:57,493 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:57,493 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:57,493 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:57,493 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:57,493 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:57,493 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:57,493 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:57,493 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. started... 2023-04-12 15:55:57,493 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. as already flushing 2023-04-12 15:55:57,493 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:57,493 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:57,493 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:57,493 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,493 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:57,493 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:57,495 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:57,495 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:57,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:57,495 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:57,495 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:57,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:57,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:57,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:57,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:57,497 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:57,497 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ca5dc2[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:57,497 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:57,497 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:57,498 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:57,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:57,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:57,498 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:57,498 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:57,498 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:57,498 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,498 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:57,500 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,500 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:57,500 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:57,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:57,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:57,501 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:57,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:57,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:57,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:57,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:57,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:57,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:57,503 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:57,503 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:57,503 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:57,503 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:57,503 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:57,503 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:57,504 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:57,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:57,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:57,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:57,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:57,505 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:57,505 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,505 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:57,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:57,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:57,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:57,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:57,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:57,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:57,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:57,507 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:57,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:57,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:57,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,509 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,509 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:57,509 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:57,509 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,509 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:57,509 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:57,509 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:57,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:57,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:57,509 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:57,509 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:57,509 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:57,509 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:57,509 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:57,509 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:57,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:57,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:57,510 DEBUG [Thread-3125] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:57,510 DEBUG [Thread-3125] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:57,566 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:57,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55260 deadline: 1681315017562, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,566 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:57,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55336 deadline: 1681315017563, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,566 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:57,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55306 deadline: 1681315017563, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,572 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:57,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681315017564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,611 DEBUG [Thread-3125] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:57,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:57,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:57,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:57,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:57,614 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:57,614 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:57,614 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:57,614 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:57,615 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:57,615 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:57,616 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,616 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:57,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:57,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:57,616 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,616 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:57,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:57,617 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=598 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/A/534031ffbd7e4e20a4804a5c05b41f7f 2023-04-12 15:55:57,617 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:57,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:57,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:57,618 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:57,618 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:57,618 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:57,618 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:57,618 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:57,618 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:57,618 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:57,619 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:57,619 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:57,619 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. started... 2023-04-12 15:55:57,619 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. as already flushing 2023-04-12 15:55:57,619 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:57,619 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:57,619 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:57,619 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,619 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:57,619 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:57,624 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:57,624 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:57,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:57,624 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:57,624 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:57,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:57,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:57,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:57,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:57,626 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:57,626 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@573ba391[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:57,626 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:57,626 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:57,627 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:57,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:57,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:57,628 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:57,628 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:57,628 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:57,628 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,628 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:57,629 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=598 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/B/3888a786764040d9bd5f4f612353a0f8 2023-04-12 15:55:57,629 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:57,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:57,629 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:57,629 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:57,629 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:57,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:57,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:57,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:57,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:57,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:57,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:57,634 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:57,634 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:57,634 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:57,634 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:57,634 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:57,634 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:57,636 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:57,636 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:57,636 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:57,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:57,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:57,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:57,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:57,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:57,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:57,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:57,637 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:57,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:57,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:57,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:57,638 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:57,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:57,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:57,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,641 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=598 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/C/3e7cc01fb1b743738a5579039c4940d3 2023-04-12 15:55:57,645 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:57,645 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:57,645 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:57,645 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:57,645 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:57,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:57,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:57,645 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:57,645 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,645 DEBUG [Thread-3125] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:57,645 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/A/534031ffbd7e4e20a4804a5c05b41f7f as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/534031ffbd7e4e20a4804a5c05b41f7f 2023-04-12 15:55:57,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:57,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:57,645 DEBUG [Thread-3125] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:57,645 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:57,646 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:57,646 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,646 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:57,646 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:57,646 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:57,649 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/534031ffbd7e4e20a4804a5c05b41f7f, entries=200, sequenceid=598, filesize=14.3 K 2023-04-12 15:55:57,650 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/B/3888a786764040d9bd5f4f612353a0f8 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/3888a786764040d9bd5f4f612353a0f8 2023-04-12 15:55:57,654 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/3888a786764040d9bd5f4f612353a0f8, entries=150, sequenceid=598, filesize=11.9 K 2023-04-12 15:55:57,654 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/C/3e7cc01fb1b743738a5579039c4940d3 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/3e7cc01fb1b743738a5579039c4940d3 2023-04-12 15:55:57,662 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/3e7cc01fb1b743738a5579039c4940d3, entries=150, sequenceid=598, filesize=11.9 K 2023-04-12 15:55:57,662 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 4df381dfb8e2792d3aca65faf2673530 in 452ms, sequenceid=598, compaction requested=true 2023-04-12 15:55:57,662 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4df381dfb8e2792d3aca65faf2673530: 2023-04-12 15:55:57,663 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 15:55:57,663 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:55:57,663 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 15:55:57,663 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:55:57,663 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:55:57,663 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40292 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 15:55:57,663 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37852 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 15:55:57,664 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1912): 4df381dfb8e2792d3aca65faf2673530/A is initiating minor compaction (all files) 2023-04-12 15:55:57,664 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1912): 4df381dfb8e2792d3aca65faf2673530/B is initiating minor compaction (all files) 2023-04-12 15:55:57,664 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4df381dfb8e2792d3aca65faf2673530/A in TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:57,664 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4df381dfb8e2792d3aca65faf2673530/B in TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:57,664 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/f02f5fcee75749cc88ccdece827a3005, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/5b4f058ea912496f93ca28483cd512a3, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/534031ffbd7e4e20a4804a5c05b41f7f] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp, totalSize=39.3 K 2023-04-12 15:55:57,664 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/44b25aaec44847d8a77f7c0214e88fea, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/fe4dda7a758c42f8a1e79edbed7a0a00, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/3888a786764040d9bd5f4f612353a0f8] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp, totalSize=37.0 K 2023-04-12 15:55:57,664 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting f02f5fcee75749cc88ccdece827a3005, keycount=150, bloomtype=ROW, size=13.2 K, encoding=NONE, compression=NONE, seqNum=562, earliestPutTs=1681314954593 2023-04-12 15:55:57,666 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 44b25aaec44847d8a77f7c0214e88fea, keycount=150, bloomtype=ROW, size=13.2 K, encoding=NONE, compression=NONE, seqNum=562, earliestPutTs=1681314954593 2023-04-12 15:55:57,666 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 5b4f058ea912496f93ca28483cd512a3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=584, earliestPutTs=1681314955750 2023-04-12 15:55:57,666 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting fe4dda7a758c42f8a1e79edbed7a0a00, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=584, earliestPutTs=1681314955750 2023-04-12 15:55:57,666 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 534031ffbd7e4e20a4804a5c05b41f7f, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=598, earliestPutTs=1681314956889 2023-04-12 15:55:57,667 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 3888a786764040d9bd5f4f612353a0f8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=598, earliestPutTs=1681314956889 2023-04-12 15:55:57,677 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4df381dfb8e2792d3aca65faf2673530#A#compaction#520 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:55:57,679 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] throttle.PressureAwareThroughputController(145): 4df381dfb8e2792d3aca65faf2673530#B#compaction#521 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:55:57,722 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/B/ae707672897640efbc2509fc8c73cbaa as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/ae707672897640efbc2509fc8c73cbaa 2023-04-12 15:55:57,726 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4df381dfb8e2792d3aca65faf2673530/B of 4df381dfb8e2792d3aca65faf2673530 into ae707672897640efbc2509fc8c73cbaa(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:55:57,726 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4df381dfb8e2792d3aca65faf2673530: 2023-04-12 15:55:57,726 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530., storeName=4df381dfb8e2792d3aca65faf2673530/B, priority=13, startTime=1681314957663; duration=0sec 2023-04-12 15:55:57,726 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:55:57,726 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 15:55:57,728 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37818 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 15:55:57,728 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1912): 4df381dfb8e2792d3aca65faf2673530/C is initiating minor compaction (all files) 2023-04-12 15:55:57,729 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4df381dfb8e2792d3aca65faf2673530/C in TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:57,729 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/a491231d287c492e888737e4602c64e9, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/adc2d7bba1154a52889909aa0a14cca5, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/3e7cc01fb1b743738a5579039c4940d3] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp, totalSize=36.9 K 2023-04-12 15:55:57,729 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting a491231d287c492e888737e4602c64e9, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=562, earliestPutTs=1681314954593 2023-04-12 15:55:57,729 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting adc2d7bba1154a52889909aa0a14cca5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=584, earliestPutTs=1681314955750 2023-04-12 15:55:57,730 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 3e7cc01fb1b743738a5579039c4940d3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=598, earliestPutTs=1681314956889 2023-04-12 15:55:57,735 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] throttle.PressureAwareThroughputController(145): 4df381dfb8e2792d3aca65faf2673530#C#compaction#522 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:55:57,746 DEBUG [Thread-3125] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:57,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:57,748 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/C/da5112797d2a45db884098f2c1b637f9 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/da5112797d2a45db884098f2c1b637f9 2023-04-12 15:55:57,749 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:57,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:57,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:57,750 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:57,750 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:57,750 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:57,750 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:57,751 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:57,751 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:57,753 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:57,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:57,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:57,753 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:57,753 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,753 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:57,753 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:57,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:57,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:57,754 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:57,754 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4df381dfb8e2792d3aca65faf2673530/C of 4df381dfb8e2792d3aca65faf2673530 into da5112797d2a45db884098f2c1b637f9(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:55:57,754 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4df381dfb8e2792d3aca65faf2673530: 2023-04-12 15:55:57,754 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:57,754 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:57,754 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530., storeName=4df381dfb8e2792d3aca65faf2673530/C, priority=13, startTime=1681314957663; duration=0sec 2023-04-12 15:55:57,754 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:57,754 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:57,754 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:57,754 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:55:57,756 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:57,756 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:57,756 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:57,756 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. started... 2023-04-12 15:55:57,756 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4df381dfb8e2792d3aca65faf2673530 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-12 15:55:57,756 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=A 2023-04-12 15:55:57,756 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:57,756 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=B 2023-04-12 15:55:57,756 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:57,756 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=C 2023-04-12 15:55:57,756 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:57,789 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=619 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/A/ac125cec6e73480bb2f1c0a16875d395 2023-04-12 15:55:57,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on 4df381dfb8e2792d3aca65faf2673530 2023-04-12 15:55:57,872 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. as already flushing 2023-04-12 15:55:57,906 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:57,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55336 deadline: 1681315017897, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,906 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:57,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55306 deadline: 1681315017897, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,907 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:57,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55260 deadline: 1681315017897, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:57,907 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:57,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681315017898, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:58,009 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:58,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55306 deadline: 1681315018007, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:58,010 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:58,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681315018008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:58,010 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:58,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55336 deadline: 1681315018008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:58,015 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:58,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55260 deadline: 1681315018008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:58,108 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/A/8a8619a4d75d45adb897bf20ea181a7e as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/8a8619a4d75d45adb897bf20ea181a7e 2023-04-12 15:55:58,112 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4df381dfb8e2792d3aca65faf2673530/A of 4df381dfb8e2792d3aca65faf2673530 into 8a8619a4d75d45adb897bf20ea181a7e(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:55:58,113 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4df381dfb8e2792d3aca65faf2673530: 2023-04-12 15:55:58,113 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530., storeName=4df381dfb8e2792d3aca65faf2673530/A, priority=13, startTime=1681314957662; duration=0sec 2023-04-12 15:55:58,113 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:55:58,118 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:58,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55340 deadline: 1681315018110, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:58,120 DEBUG [Thread-3117] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4165 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530., hostname=jenkins-hbase4.apache.org,38905,1681314818739, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-12 15:55:58,202 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=619 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/B/eb545342586f4371ae4f99e266397212 2023-04-12 15:55:58,215 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:58,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55336 deadline: 1681315018211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:58,215 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:58,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55306 deadline: 1681315018211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:58,215 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:58,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681315018212, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:58,220 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:58,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55260 deadline: 1681315018216, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:58,521 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:58,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681315018517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:58,522 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:58,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55336 deadline: 1681315018518, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:58,522 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:58,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55306 deadline: 1681315018518, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:58,526 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:58,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55260 deadline: 1681315018523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:58,615 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=619 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/C/303ae31417c0418fb2f04e4b817913d7 2023-04-12 15:55:58,622 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/A/ac125cec6e73480bb2f1c0a16875d395 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/ac125cec6e73480bb2f1c0a16875d395 2023-04-12 15:55:58,625 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/ac125cec6e73480bb2f1c0a16875d395, entries=150, sequenceid=619, filesize=11.9 K 2023-04-12 15:55:58,626 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/B/eb545342586f4371ae4f99e266397212 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/eb545342586f4371ae4f99e266397212 2023-04-12 15:55:58,629 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/eb545342586f4371ae4f99e266397212, entries=150, sequenceid=619, filesize=11.9 K 2023-04-12 15:55:58,630 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/C/303ae31417c0418fb2f04e4b817913d7 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/303ae31417c0418fb2f04e4b817913d7 2023-04-12 15:55:58,633 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/303ae31417c0418fb2f04e4b817913d7, entries=150, sequenceid=619, filesize=11.9 K 2023-04-12 15:55:58,634 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=73.80 KB/75570 for 4df381dfb8e2792d3aca65faf2673530 in 877ms, sequenceid=619, compaction requested=false 2023-04-12 15:55:58,634 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4df381dfb8e2792d3aca65faf2673530: 2023-04-12 15:55:58,634 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:58,634 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:58,634 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:58,634 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:58,634 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:58,634 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:58,635 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:58,635 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:58,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:58,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:58,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:58,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:58,636 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:58,636 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:58,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:58,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:58,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:58,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:58,637 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:58,637 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d6164ca[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:58,637 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:58,637 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:58,638 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:58,638 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:58,639 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:58,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:58,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:58,639 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:58,639 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:58,639 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:58,642 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:58,642 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:58,642 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:58,642 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:58,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:58,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:58,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:58,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:58,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:58,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:58,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:58,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:58,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:58,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:58,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:58,644 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:58,645 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:58,645 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:58,645 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:58,645 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:58,645 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:58,646 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:58,646 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:58,646 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:58,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:58,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:58,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:58,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:58,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:58,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:58,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:58,647 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:58,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:58,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:58,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:58,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:58,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:58,648 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:58,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:58,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:58,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:58,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:58,651 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:58,651 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:58,651 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:58,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:58,651 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:58,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:58,651 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:58,651 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:58,651 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:58,651 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:58,651 DEBUG [Thread-3125] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:58,652 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:58,652 DEBUG [Thread-3125] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:58,652 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:58,652 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:58,652 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:58,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:58,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:58,654 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:58,752 DEBUG [Thread-3125] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:58,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:58,754 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:58,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:58,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:58,755 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:58,755 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:58,755 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:58,755 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:58,756 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:58,756 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:58,757 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:58,757 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:58,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:58,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:58,757 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:58,757 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:58,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:58,758 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:58,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:58,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:58,758 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:58,758 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:58,758 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:58,759 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:58,759 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:58,759 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:58,759 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:58,759 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:58,759 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:58,759 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. started... 2023-04-12 15:55:58,759 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4df381dfb8e2792d3aca65faf2673530 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-12 15:55:58,759 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=A 2023-04-12 15:55:58,759 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:58,759 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=B 2023-04-12 15:55:58,760 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:58,760 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=C 2023-04-12 15:55:58,760 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:58,768 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=634 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/A/78ea83de5dca46358001967218decd03 2023-04-12 15:55:58,782 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=634 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/B/c9227152399a45f89ea6c68ea178533d 2023-04-12 15:55:58,806 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=634 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/C/e131c346d6bd460b9abdc588605fb936 2023-04-12 15:55:58,810 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/A/78ea83de5dca46358001967218decd03 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/78ea83de5dca46358001967218decd03 2023-04-12 15:55:58,814 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/78ea83de5dca46358001967218decd03, entries=150, sequenceid=634, filesize=11.9 K 2023-04-12 15:55:58,815 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/B/c9227152399a45f89ea6c68ea178533d as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/c9227152399a45f89ea6c68ea178533d 2023-04-12 15:55:58,818 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/c9227152399a45f89ea6c68ea178533d, entries=150, sequenceid=634, filesize=11.9 K 2023-04-12 15:55:58,819 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/C/e131c346d6bd460b9abdc588605fb936 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/e131c346d6bd460b9abdc588605fb936 2023-04-12 15:55:58,823 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/e131c346d6bd460b9abdc588605fb936, entries=150, sequenceid=634, filesize=11.9 K 2023-04-12 15:55:58,824 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=0 B/0 for 4df381dfb8e2792d3aca65faf2673530 in 65ms, sequenceid=634, compaction requested=true 2023-04-12 15:55:58,824 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4df381dfb8e2792d3aca65faf2673530: 2023-04-12 15:55:58,824 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:58,824 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:58,824 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:58,824 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:58,824 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:58,824 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:58,826 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:58,826 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:58,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:58,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:58,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:58,826 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:58,826 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:58,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:58,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:58,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:58,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:58,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:58,828 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:58,828 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40382251[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:58,828 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:58,828 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:58,830 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:58,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:58,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:58,830 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:58,830 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:58,830 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:58,830 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:58,830 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:58,831 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:58,831 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:58,831 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:58,832 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:58,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:58,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:58,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:58,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:58,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:58,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:58,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:58,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:58,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:58,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:58,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:58,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:58,838 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:58,838 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:58,838 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:58,838 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:58,838 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:58,839 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:58,839 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:58,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:58,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:58,839 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:58,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:58,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:58,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:58,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:58,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:58,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:58,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:58,841 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:58,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:58,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:58,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:58,843 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:58,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:58,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:58,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:58,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:58,846 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:58,846 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:58,846 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:58,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:58,846 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:58,846 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:58,846 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:58,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:58,846 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:58,846 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:58,846 DEBUG [Thread-3125] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:58,846 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:58,846 DEBUG [Thread-3125] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:58,846 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:58,846 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:58,846 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:58,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:58,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:58,846 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:58,946 DEBUG [Thread-3125] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:58,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:58,949 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:58,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:58,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:58,950 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:58,950 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:58,950 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:58,950 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:58,951 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:58,951 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:58,952 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:58,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:58,952 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:58,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:58,953 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:58,953 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:58,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:58,953 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:58,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:58,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:58,953 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:58,954 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:58,954 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:58,954 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:58,954 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:58,954 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:58,954 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:58,954 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:58,954 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:58,954 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. started... 2023-04-12 15:55:58,954 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4df381dfb8e2792d3aca65faf2673530: 2023-04-12 15:55:58,954 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:58,954 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:58,954 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:58,954 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:58,954 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:58,955 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:58,956 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:58,956 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:58,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:58,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:58,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:58,957 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:58,957 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:58,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:58,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:58,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:58,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:58,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:58,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:58,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@174b53d9[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:58,958 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:58,958 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:58,960 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:58,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:58,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:58,960 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:58,960 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:58,960 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:58,960 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:58,960 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:58,962 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:58,962 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:58,962 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:58,962 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:58,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:58,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:58,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:58,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:58,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:58,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:58,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:58,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:58,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:58,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:58,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:58,965 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:58,965 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:58,965 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:58,965 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:58,965 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:58,965 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:58,966 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:58,966 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:58,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:58,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:58,966 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:58,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:58,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:58,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:58,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:58,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:58,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:58,967 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:58,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:58,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:58,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:58,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:58,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:58,969 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:58,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:58,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:58,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:58,978 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:58,978 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:58,978 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:58,978 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:58,978 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:58,978 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:58,978 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:58,978 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:58,978 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:58,978 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:58,978 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:58,978 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:58,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:58,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:58,979 DEBUG [Thread-3125] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:58,979 DEBUG [Thread-3125] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:58,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:58,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:59,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on 4df381dfb8e2792d3aca65faf2673530 2023-04-12 15:55:59,035 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4df381dfb8e2792d3aca65faf2673530 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-12 15:55:59,035 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=A 2023-04-12 15:55:59,035 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:59,035 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=B 2023-04-12 15:55:59,035 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:59,035 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=C 2023-04-12 15:55:59,035 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:55:59,079 DEBUG [Thread-3125] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:59,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:59,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:59,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:59,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:59,081 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:59,081 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:59,081 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:59,081 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:59,082 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:59,082 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:59,084 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:59,084 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:59,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:59,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:59,084 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:59,084 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:59,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:59,085 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:59,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:59,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:59,085 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:59,085 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:59,085 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:59,085 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:59,085 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:59,085 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:59,086 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:59,086 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:59,086 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:59,086 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. started... 2023-04-12 15:55:59,086 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. as already flushing 2023-04-12 15:55:59,086 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:59,086 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:59,086 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:59,086 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:59,086 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:59,086 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:59,088 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:59,088 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:59,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:59,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:59,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:59,088 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:59,088 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:59,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:59,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:59,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:59,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:59,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:59,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:59,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d664d8a[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:59,089 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:59,089 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:59,090 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:59,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:59,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:59,091 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:59,091 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:59,091 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:59,091 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:59,091 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:59,093 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:59,093 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:59,093 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:59,093 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:59,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:59,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:59,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:59,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:59,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:59,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:59,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:59,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:59,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:59,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:59,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:59,095 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:59,095 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:59,095 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:59,096 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:59,096 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:59,096 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:59,098 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:59,098 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:59,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:59,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:59,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:59,099 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:59,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:59,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:59,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:59,099 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:59,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:59,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:59,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:59,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:59,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:59,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:59,101 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:59,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:59,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:59,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:59,102 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681314959083, max timestamp=1681314959100Num uniques 859; 2023-04-12 15:55:59,102 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-12 15:55:59,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:59,104 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681314959083, max timestamp=1681314959100Num uniques 859; 2023-04-12 15:55:59,104 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-12 15:55:59,104 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681314959083, max timestamp=1681314959100Num uniques 859; 2023-04-12 15:55:59,104 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-12 15:55:59,105 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:59,105 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:59,105 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:59,105 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:59,105 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:59,105 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:59,105 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:59,105 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:59,105 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:59,105 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:59,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:59,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:59,105 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:59,105 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:59,105 DEBUG [Thread-3125] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:59,105 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:59,105 DEBUG [Thread-3125] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:59,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:59,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:59,146 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:59,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55306 deadline: 1681315019127, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:59,147 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:59,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55260 deadline: 1681315019135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:59,151 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:59,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55336 deadline: 1681315019138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:59,152 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:59,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681315019138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:59,206 DEBUG [Thread-3125] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:59,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:59,208 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:59,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:59,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:59,210 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:59,210 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:59,210 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:59,210 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:59,210 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:59,210 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:59,212 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:59,212 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:59,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:59,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:59,212 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:59,212 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:59,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:59,212 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:59,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:59,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:59,213 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:59,213 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:59,213 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:59,213 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:59,213 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:59,213 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:59,213 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:59,213 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:59,214 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:59,214 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. started... 2023-04-12 15:55:59,214 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. as already flushing 2023-04-12 15:55:59,214 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:59,214 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:59,214 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:59,214 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:59,214 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:59,214 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:59,215 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:59,216 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:59,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:59,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:59,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:59,216 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:59,216 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:59,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:59,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:59,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:59,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:59,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:59,217 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:59,217 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b772dd8[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:59,217 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:59,217 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:59,222 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:59,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:59,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:59,222 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:59,222 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:59,222 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:59,224 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:59,224 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:59,224 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:59,224 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:59,225 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:59,225 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:59,225 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:59,225 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:59,225 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:59,225 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:59,225 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:59,226 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:59,226 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:59,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:59,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:59,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:59,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:59,226 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:59,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:59,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:59,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:59,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:59,227 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:59,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:59,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:59,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:59,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:59,229 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:59,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:59,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:59,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:59,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:59,231 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:59,232 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:59,232 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:59,232 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:59,232 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:59,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:59,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:59,232 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:59,232 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:59,232 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:59,232 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:59,232 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:59,232 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:59,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:59,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:59,232 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:59,232 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:59,232 DEBUG [Thread-3125] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:59,232 DEBUG [Thread-3125] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:59,253 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:59,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55306 deadline: 1681315019247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:59,255 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:59,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55260 deadline: 1681315019248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:59,256 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:59,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55336 deadline: 1681315019252, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:59,256 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:59,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681315019253, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:59,333 DEBUG [Thread-3125] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:59,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:59,335 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:59,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:59,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:59,336 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:59,336 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:59,336 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:59,336 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:59,337 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:59,337 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:59,339 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:59,339 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:59,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:59,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:59,339 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:59,339 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:59,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:59,340 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:59,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:59,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:59,340 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:59,340 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:59,341 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:59,341 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:59,341 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:59,341 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:59,341 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:59,341 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:59,341 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:59,341 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. started... 2023-04-12 15:55:59,341 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. as already flushing 2023-04-12 15:55:59,341 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:59,341 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:59,341 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:59,341 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:59,341 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:59,341 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:59,343 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:59,343 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:59,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:59,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:59,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:59,344 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:59,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:59,344 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:59,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:59,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:59,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:59,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:59,345 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:59,345 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a5cb44d[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:59,345 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:59,345 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:59,348 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:59,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:59,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:59,348 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:59,348 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:59,348 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:59,348 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:59,348 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:59,349 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:59,349 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:59,349 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:59,349 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:59,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:59,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:59,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:59,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:59,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:59,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:59,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:59,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:59,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:59,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:59,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:59,353 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:59,353 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:59,353 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:59,353 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:59,353 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:59,353 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:59,357 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:59,357 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:59,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:59,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:59,357 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:59,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:59,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:59,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:59,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:59,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:59,358 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:59,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:59,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:59,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:59,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:59,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:59,359 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:59,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:59,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:59,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:59,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:59,362 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:59,362 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:59,362 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:59,362 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:59,362 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:59,362 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:59,362 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:59,362 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:59,362 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:59,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:59,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:59,362 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:59,362 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:59,362 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:59,362 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:59,362 DEBUG [Thread-3125] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:59,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:59,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:59,362 DEBUG [Thread-3125] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:59,447 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=646 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/A/4521f4fd7e514ab092bac4e2c1747b13 2023-04-12 15:55:59,458 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=646 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/B/ea3c1d04f38d41ff8ba7ef560e460d50 2023-04-12 15:55:59,462 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:59,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55306 deadline: 1681315019455, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:59,462 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:59,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55260 deadline: 1681315019457, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:59,463 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:59,463 DEBUG [Thread-3125] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:59,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681315019457, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:59,463 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:59,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55336 deadline: 1681315019458, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:59,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:59,465 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:59,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:59,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:59,466 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:59,466 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:59,466 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:59,466 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:59,467 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:59,467 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:59,470 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:59,470 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:59,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:59,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:59,471 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:59,471 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:59,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:59,471 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:59,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:59,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:59,471 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:59,477 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:59,477 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:59,477 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:59,477 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:59,477 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:59,478 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:59,478 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:59,478 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:59,478 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. started... 2023-04-12 15:55:59,479 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. as already flushing 2023-04-12 15:55:59,479 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:59,479 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:55:59,479 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:55:59,479 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:59,479 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:55:59,479 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:55:59,479 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=646 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/C/3da6a04c762c4994be6fb21b2c57aa4c 2023-04-12 15:55:59,482 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:59,482 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:59,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:59,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:59,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:59,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:59,483 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:59,483 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:55:59,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:59,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:59,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:59,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:59,484 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:55:59,484 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@744e344[Count = 0] remaining members to acquire global barrier 2023-04-12 15:55:59,484 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:55:59,484 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:59,484 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/A/4521f4fd7e514ab092bac4e2c1747b13 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/4521f4fd7e514ab092bac4e2c1747b13 2023-04-12 15:55:59,486 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:59,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:59,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:59,486 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:55:59,486 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:55:59,486 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:55:59,487 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:59,487 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:55:59,488 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:55:59,488 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:59,489 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:59,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:59,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:59,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:59,489 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:55:59,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:59,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:59,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:59,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:59,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:59,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:59,491 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/4521f4fd7e514ab092bac4e2c1747b13, entries=300, sequenceid=646, filesize=19.0 K 2023-04-12 15:55:59,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:59,492 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/B/ea3c1d04f38d41ff8ba7ef560e460d50 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/ea3c1d04f38d41ff8ba7ef560e460d50 2023-04-12 15:55:59,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:55:59,492 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:55:59,492 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:55:59,492 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:55:59,492 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:55:59,492 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:55:59,492 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:55:59,494 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:59,494 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:59,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:59,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:59,494 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:59,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:59,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:55:59,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:55:59,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:59,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:59,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:55:59,495 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:59,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:59,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:59,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:55:59,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:59,496 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:59,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:59,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:55:59,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:55:59,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:59,499 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/ea3c1d04f38d41ff8ba7ef560e460d50, entries=150, sequenceid=646, filesize=11.9 K 2023-04-12 15:55:59,499 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:59,499 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:55:59,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:59,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:59,500 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:59,500 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:55:59,500 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:55:59,500 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/C/3da6a04c762c4994be6fb21b2c57aa4c as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/3da6a04c762c4994be6fb21b2c57aa4c 2023-04-12 15:55:59,500 DEBUG [Thread-3125] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:55:59,500 DEBUG [Thread-3125] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:55:59,500 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:59,500 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:59,500 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:59,500 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:59,500 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:55:59,500 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:59,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:55:59,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:55:59,500 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:55:59,500 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:55:59,505 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/3da6a04c762c4994be6fb21b2c57aa4c, entries=150, sequenceid=646, filesize=11.9 K 2023-04-12 15:55:59,506 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 4df381dfb8e2792d3aca65faf2673530 in 472ms, sequenceid=646, compaction requested=true 2023-04-12 15:55:59,506 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4df381dfb8e2792d3aca65faf2673530: 2023-04-12 15:55:59,506 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-12 15:55:59,507 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:55:59,507 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:55:59,507 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:55:59,507 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-12 15:55:59,508 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 57460 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-12 15:55:59,508 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 50140 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-12 15:55:59,508 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1912): 4df381dfb8e2792d3aca65faf2673530/A is initiating minor compaction (all files) 2023-04-12 15:55:59,508 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1912): 4df381dfb8e2792d3aca65faf2673530/B is initiating minor compaction (all files) 2023-04-12 15:55:59,508 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4df381dfb8e2792d3aca65faf2673530/A in TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:59,508 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4df381dfb8e2792d3aca65faf2673530/B in TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:59,508 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/8a8619a4d75d45adb897bf20ea181a7e, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/ac125cec6e73480bb2f1c0a16875d395, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/78ea83de5dca46358001967218decd03, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/4521f4fd7e514ab092bac4e2c1747b13] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp, totalSize=56.1 K 2023-04-12 15:55:59,508 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/ae707672897640efbc2509fc8c73cbaa, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/eb545342586f4371ae4f99e266397212, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/c9227152399a45f89ea6c68ea178533d, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/ea3c1d04f38d41ff8ba7ef560e460d50] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp, totalSize=49.0 K 2023-04-12 15:55:59,509 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 8a8619a4d75d45adb897bf20ea181a7e, keycount=150, bloomtype=ROW, size=13.3 K, encoding=NONE, compression=NONE, seqNum=598, earliestPutTs=1681314956889 2023-04-12 15:55:59,509 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting ae707672897640efbc2509fc8c73cbaa, keycount=150, bloomtype=ROW, size=13.3 K, encoding=NONE, compression=NONE, seqNum=598, earliestPutTs=1681314956889 2023-04-12 15:55:59,509 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting ac125cec6e73480bb2f1c0a16875d395, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=619, earliestPutTs=1681314957229 2023-04-12 15:55:59,509 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting eb545342586f4371ae4f99e266397212, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=619, earliestPutTs=1681314957229 2023-04-12 15:55:59,509 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 78ea83de5dca46358001967218decd03, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=634, earliestPutTs=1681314957878 2023-04-12 15:55:59,510 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting c9227152399a45f89ea6c68ea178533d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=634, earliestPutTs=1681314957878 2023-04-12 15:55:59,510 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 4521f4fd7e514ab092bac4e2c1747b13, keycount=300, bloomtype=ROW, size=19.0 K, encoding=NONE, compression=NONE, seqNum=646, earliestPutTs=1681314959028 2023-04-12 15:55:59,510 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting ea3c1d04f38d41ff8ba7ef560e460d50, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=646, earliestPutTs=1681314959033 2023-04-12 15:55:59,519 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4df381dfb8e2792d3aca65faf2673530#A#compaction#532 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:55:59,524 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] throttle.PressureAwareThroughputController(145): 4df381dfb8e2792d3aca65faf2673530#B#compaction#533 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:55:59,541 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/B/00645d88478341d0828aa8ac26778a6b as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/00645d88478341d0828aa8ac26778a6b 2023-04-12 15:55:59,545 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 4df381dfb8e2792d3aca65faf2673530/B of 4df381dfb8e2792d3aca65faf2673530 into 00645d88478341d0828aa8ac26778a6b(size=13.4 K), total size for store is 13.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:55:59,545 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4df381dfb8e2792d3aca65faf2673530: 2023-04-12 15:55:59,545 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530., storeName=4df381dfb8e2792d3aca65faf2673530/B, priority=12, startTime=1681314959507; duration=0sec 2023-04-12 15:55:59,545 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:55:59,545 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-12 15:55:59,547 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 50106 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-12 15:55:59,547 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1912): 4df381dfb8e2792d3aca65faf2673530/C is initiating minor compaction (all files) 2023-04-12 15:55:59,547 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4df381dfb8e2792d3aca65faf2673530/C in TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:59,547 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/da5112797d2a45db884098f2c1b637f9, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/303ae31417c0418fb2f04e4b817913d7, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/e131c346d6bd460b9abdc588605fb936, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/3da6a04c762c4994be6fb21b2c57aa4c] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp, totalSize=48.9 K 2023-04-12 15:55:59,547 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting da5112797d2a45db884098f2c1b637f9, keycount=150, bloomtype=ROW, size=13.2 K, encoding=NONE, compression=NONE, seqNum=598, earliestPutTs=1681314956889 2023-04-12 15:55:59,548 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 303ae31417c0418fb2f04e4b817913d7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=619, earliestPutTs=1681314957229 2023-04-12 15:55:59,548 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting e131c346d6bd460b9abdc588605fb936, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=634, earliestPutTs=1681314957878 2023-04-12 15:55:59,548 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 3da6a04c762c4994be6fb21b2c57aa4c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=646, earliestPutTs=1681314959033 2023-04-12 15:55:59,554 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] throttle.PressureAwareThroughputController(145): 4df381dfb8e2792d3aca65faf2673530#C#compaction#534 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:55:59,564 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/C/757c9ec80ddc4ff7b9732ccbabd4108d as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/757c9ec80ddc4ff7b9732ccbabd4108d 2023-04-12 15:55:59,569 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 4df381dfb8e2792d3aca65faf2673530/C of 4df381dfb8e2792d3aca65faf2673530 into 757c9ec80ddc4ff7b9732ccbabd4108d(size=13.4 K), total size for store is 13.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:55:59,569 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4df381dfb8e2792d3aca65faf2673530: 2023-04-12 15:55:59,569 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530., storeName=4df381dfb8e2792d3aca65faf2673530/C, priority=12, startTime=1681314959507; duration=0sec 2023-04-12 15:55:59,569 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:55:59,600 DEBUG [Thread-3125] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:55:59,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:55:59,602 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:55:59,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:55:59,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:55:59,603 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:55:59,603 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:59,603 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:55:59,603 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:55:59,604 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:59,604 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:59,605 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:55:59,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:55:59,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:55:59,605 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:59,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:59,605 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:59,605 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:55:59,606 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:55:59,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:55:59,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:55:59,606 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:55:59,607 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:55:59,607 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:55:59,607 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:55:59,607 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:55:59,607 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:55:59,608 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:55:59,608 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:55:59,608 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:55:59,608 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. started... 2023-04-12 15:55:59,608 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4df381dfb8e2792d3aca65faf2673530 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-12 15:55:59,608 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=A 2023-04-12 15:55:59,608 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:55:59,608 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=B 2023-04-12 15:55:59,608 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:55:59,608 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=C 2023-04-12 15:55:59,608 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:55:59,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on 4df381dfb8e2792d3aca65faf2673530 2023-04-12 15:55:59,768 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. as already flushing 2023-04-12 15:55:59,786 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:59,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55306 deadline: 1681315019780, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:59,787 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:59,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681315019781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:59,787 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:59,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 277 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55260 deadline: 1681315019781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:59,787 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:59,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55336 deadline: 1681315019782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:59,894 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:59,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55306 deadline: 1681315019887, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:59,894 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:59,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681315019887, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:59,894 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:59,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 279 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55260 deadline: 1681315019888, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:59,894 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:55:59,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55336 deadline: 1681315019888, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:55:59,935 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/A/ec2dc32db6494cfb8be5411acb56c307 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/ec2dc32db6494cfb8be5411acb56c307 2023-04-12 15:55:59,939 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 4df381dfb8e2792d3aca65faf2673530/A of 4df381dfb8e2792d3aca65faf2673530 into ec2dc32db6494cfb8be5411acb56c307(size=13.4 K), total size for store is 13.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:55:59,939 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4df381dfb8e2792d3aca65faf2673530: 2023-04-12 15:55:59,939 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530., storeName=4df381dfb8e2792d3aca65faf2673530/A, priority=12, startTime=1681314959506; duration=0sec 2023-04-12 15:55:59,939 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:56:00,021 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=683 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/A/adbe50515474414f803438c43deaeaae 2023-04-12 15:56:00,098 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:00,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55336 deadline: 1681315020095, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:00,098 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:00,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55306 deadline: 1681315020096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:00,100 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:00,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681315020096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:00,100 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:00,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 281 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55260 deadline: 1681315020096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:00,404 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:00,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55336 deadline: 1681315020399, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:00,404 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:00,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55306 deadline: 1681315020400, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:00,404 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:00,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681315020401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:00,405 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:00,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 283 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55260 deadline: 1681315020401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:00,434 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=683 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/B/2ca974a5ed6647af8e1af7b263bc7815 2023-04-12 15:56:00,446 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=683 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/C/3b96fea49487436a8e8d79d6de90772b 2023-04-12 15:56:00,451 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/A/adbe50515474414f803438c43deaeaae as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/adbe50515474414f803438c43deaeaae 2023-04-12 15:56:00,458 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/adbe50515474414f803438c43deaeaae, entries=150, sequenceid=683, filesize=11.9 K 2023-04-12 15:56:00,459 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/B/2ca974a5ed6647af8e1af7b263bc7815 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/2ca974a5ed6647af8e1af7b263bc7815 2023-04-12 15:56:00,463 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/2ca974a5ed6647af8e1af7b263bc7815, entries=150, sequenceid=683, filesize=11.9 K 2023-04-12 15:56:00,464 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/C/3b96fea49487436a8e8d79d6de90772b as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/3b96fea49487436a8e8d79d6de90772b 2023-04-12 15:56:00,466 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/3b96fea49487436a8e8d79d6de90772b, entries=150, sequenceid=683, filesize=11.9 K 2023-04-12 15:56:00,467 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for 4df381dfb8e2792d3aca65faf2673530 in 859ms, sequenceid=683, compaction requested=false 2023-04-12 15:56:00,467 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4df381dfb8e2792d3aca65faf2673530: 2023-04-12 15:56:00,467 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:56:00,467 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:56:00,467 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:56:00,467 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:00,467 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:56:00,467 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:56:00,469 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:00,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:00,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:00,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:00,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:00,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:00,470 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:00,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:00,470 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:00,470 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:56:00,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:00,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:00,471 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:56:00,471 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28e1f43b[Count = 0] remaining members to acquire global barrier 2023-04-12 15:56:00,471 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:56:00,471 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:00,473 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:00,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:00,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:00,473 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:56:00,473 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:56:00,473 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:56:00,473 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:00,473 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:56:00,475 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:56:00,475 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:00,475 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:00,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:00,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:00,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:00,475 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:56:00,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:00,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:00,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:00,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:00,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:00,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:00,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:00,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:56:00,477 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:56:00,477 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:56:00,477 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:56:00,477 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:56:00,477 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:56:00,477 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:56:00,479 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:00,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:00,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:00,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:00,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:00,480 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:00,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:00,480 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:00,480 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:00,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:00,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:00,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:00,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:00,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:00,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:00,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:00,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:00,482 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:00,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:00,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:00,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:00,486 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:00,486 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:00,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:00,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:00,486 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:00,486 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:00,486 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:00,486 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:00,486 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:00,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:00,486 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:00,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:00,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:56:00,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:56:00,486 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:00,487 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:00,487 DEBUG [Thread-3125] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:56:00,487 DEBUG [Thread-3125] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:56:00,587 DEBUG [Thread-3125] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:56:00,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:56:00,589 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:56:00,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:56:00,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:56:00,590 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:56:00,590 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:00,590 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:56:00,590 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:56:00,591 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:00,591 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:00,592 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:00,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:00,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:00,592 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:00,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:00,593 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:00,593 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:56:00,593 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:00,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:56:00,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:00,593 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:56:00,594 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:56:00,594 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:56:00,594 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:56:00,594 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:00,594 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:56:00,594 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:56:00,594 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:56:00,594 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:56:00,594 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. started... 2023-04-12 15:56:00,594 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4df381dfb8e2792d3aca65faf2673530 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-12 15:56:00,595 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=A 2023-04-12 15:56:00,595 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:00,595 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=B 2023-04-12 15:56:00,595 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:00,595 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=C 2023-04-12 15:56:00,595 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:00,604 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=695 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/A/e418f998054b4fd4b11787a4b8f91ed6 2023-04-12 15:56:00,621 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=695 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/B/b6c9129c55254f95bc20c3b5d8e5e584 2023-04-12 15:56:00,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on 4df381dfb8e2792d3aca65faf2673530 2023-04-12 15:56:00,907 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. as already flushing 2023-04-12 15:56:00,933 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:00,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55336 deadline: 1681315020931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:00,934 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681314960914, max timestamp=1681314960932Num uniques 859; 2023-04-12 15:56:00,934 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681314960914, max timestamp=1681314960932Num uniques 859; 2023-04-12 15:56:00,934 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-12 15:56:00,935 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681314960914, max timestamp=1681314960932Num uniques 859; 2023-04-12 15:56:00,934 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-12 15:56:00,935 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-12 15:56:00,971 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:00,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 294 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55260 deadline: 1681315020963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:00,972 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:00,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 274 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681315020964, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:00,974 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:00,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 275 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55306 deadline: 1681315020966, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:01,037 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:01,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55336 deadline: 1681315021036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:01,043 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=695 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/C/d17b9d8b01cc40aea7b330068885b9bc 2023-04-12 15:56:01,047 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/A/e418f998054b4fd4b11787a4b8f91ed6 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/e418f998054b4fd4b11787a4b8f91ed6 2023-04-12 15:56:01,051 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/e418f998054b4fd4b11787a4b8f91ed6, entries=150, sequenceid=695, filesize=11.9 K 2023-04-12 15:56:01,052 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/B/b6c9129c55254f95bc20c3b5d8e5e584 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/b6c9129c55254f95bc20c3b5d8e5e584 2023-04-12 15:56:01,056 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/b6c9129c55254f95bc20c3b5d8e5e584, entries=150, sequenceid=695, filesize=11.9 K 2023-04-12 15:56:01,057 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/C/d17b9d8b01cc40aea7b330068885b9bc as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/d17b9d8b01cc40aea7b330068885b9bc 2023-04-12 15:56:01,060 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/d17b9d8b01cc40aea7b330068885b9bc, entries=150, sequenceid=695, filesize=11.9 K 2023-04-12 15:56:01,061 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for 4df381dfb8e2792d3aca65faf2673530 in 467ms, sequenceid=695, compaction requested=true 2023-04-12 15:56:01,061 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4df381dfb8e2792d3aca65faf2673530: 2023-04-12 15:56:01,061 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:56:01,061 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:56:01,061 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:56:01,061 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:01,061 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:56:01,061 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:56:01,070 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:01,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:01,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:01,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:01,070 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:01,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:01,071 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:01,071 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:56:01,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:01,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:01,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:01,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:01,073 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:56:01,073 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a8a1512[Count = 0] remaining members to acquire global barrier 2023-04-12 15:56:01,073 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:56:01,073 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:01,075 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:01,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:01,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:01,075 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:01,075 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:56:01,076 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:56:01,076 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:56:01,076 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:56:01,077 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:56:01,077 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:01,077 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:01,077 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:56:01,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:01,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:01,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:01,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:01,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:01,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:01,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:01,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on 4df381dfb8e2792d3aca65faf2673530 2023-04-12 15:56:01,079 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4df381dfb8e2792d3aca65faf2673530 3/3 column families, dataSize=140.89 KB heapSize=390.84 KB 2023-04-12 15:56:01,080 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=A 2023-04-12 15:56:01,080 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:56:01,080 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=B 2023-04-12 15:56:01,080 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:56:01,080 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=C 2023-04-12 15:56:01,080 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:56:01,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:01,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:01,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:01,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:56:01,083 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:56:01,083 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:56:01,083 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:56:01,083 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:56:01,083 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:56:01,083 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:56:01,085 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:01,085 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:01,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:01,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:01,085 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:01,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:01,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:01,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:01,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:01,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:01,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:01,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:01,086 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:01,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:01,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:01,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:01,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:01,087 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:01,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:01,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:01,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:01,092 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:01,092 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:01,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:01,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:01,092 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:01,092 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:01,092 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:01,092 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:01,092 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:01,092 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:01,092 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:01,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:01,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:01,092 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:56:01,092 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:56:01,092 DEBUG [Thread-3125] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:56:01,093 DEBUG [Thread-3125] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:56:01,092 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:01,095 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=731 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/A/a0066a7d882e4d039d39401fb058db70 2023-04-12 15:56:01,106 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:01,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 279 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681315021093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:01,120 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:01,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 280 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55306 deadline: 1681315021107, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:01,120 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:01,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 299 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55260 deadline: 1681315021107, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:01,124 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=731 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/B/f15120f788a84f9bb0122694f09b4215 2023-04-12 15:56:01,141 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=731 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/C/83862bd5f02e4586ac96f31ac56d2000 2023-04-12 15:56:01,145 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/A/a0066a7d882e4d039d39401fb058db70 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/a0066a7d882e4d039d39401fb058db70 2023-04-12 15:56:01,150 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/a0066a7d882e4d039d39401fb058db70, entries=200, sequenceid=731, filesize=14.3 K 2023-04-12 15:56:01,150 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/B/f15120f788a84f9bb0122694f09b4215 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/f15120f788a84f9bb0122694f09b4215 2023-04-12 15:56:01,154 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/f15120f788a84f9bb0122694f09b4215, entries=150, sequenceid=731, filesize=11.9 K 2023-04-12 15:56:01,155 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/C/83862bd5f02e4586ac96f31ac56d2000 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/83862bd5f02e4586ac96f31ac56d2000 2023-04-12 15:56:01,160 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/83862bd5f02e4586ac96f31ac56d2000, entries=150, sequenceid=731, filesize=11.9 K 2023-04-12 15:56:01,161 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~410.72 KB/420576, currentSize=46.96 KB/48090 for 4df381dfb8e2792d3aca65faf2673530 in 82ms, sequenceid=731, compaction requested=true 2023-04-12 15:56:01,161 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4df381dfb8e2792d3aca65faf2673530: 2023-04-12 15:56:01,161 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:56:01,161 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-12 15:56:01,161 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-12 15:56:01,161 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:56:01,161 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:56:01,162 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 52716 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-12 15:56:01,162 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1912): 4df381dfb8e2792d3aca65faf2673530/A is initiating minor compaction (all files) 2023-04-12 15:56:01,162 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 50276 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-12 15:56:01,162 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4df381dfb8e2792d3aca65faf2673530/A in TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:56:01,162 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1912): 4df381dfb8e2792d3aca65faf2673530/B is initiating minor compaction (all files) 2023-04-12 15:56:01,163 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/ec2dc32db6494cfb8be5411acb56c307, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/adbe50515474414f803438c43deaeaae, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/e418f998054b4fd4b11787a4b8f91ed6, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/a0066a7d882e4d039d39401fb058db70] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp, totalSize=51.5 K 2023-04-12 15:56:01,163 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4df381dfb8e2792d3aca65faf2673530/B in TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:56:01,163 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/00645d88478341d0828aa8ac26778a6b, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/2ca974a5ed6647af8e1af7b263bc7815, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/b6c9129c55254f95bc20c3b5d8e5e584, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/f15120f788a84f9bb0122694f09b4215] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp, totalSize=49.1 K 2023-04-12 15:56:01,163 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting ec2dc32db6494cfb8be5411acb56c307, keycount=150, bloomtype=ROW, size=13.4 K, encoding=NONE, compression=NONE, seqNum=646, earliestPutTs=1681314959033 2023-04-12 15:56:01,163 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 00645d88478341d0828aa8ac26778a6b, keycount=150, bloomtype=ROW, size=13.4 K, encoding=NONE, compression=NONE, seqNum=646, earliestPutTs=1681314959033 2023-04-12 15:56:01,163 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting adbe50515474414f803438c43deaeaae, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=683, earliestPutTs=1681314959125 2023-04-12 15:56:01,163 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 2ca974a5ed6647af8e1af7b263bc7815, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=683, earliestPutTs=1681314959125 2023-04-12 15:56:01,164 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting e418f998054b4fd4b11787a4b8f91ed6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=695, earliestPutTs=1681314959772 2023-04-12 15:56:01,164 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting b6c9129c55254f95bc20c3b5d8e5e584, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=695, earliestPutTs=1681314959772 2023-04-12 15:56:01,164 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting a0066a7d882e4d039d39401fb058db70, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=731, earliestPutTs=1681314960962 2023-04-12 15:56:01,164 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting f15120f788a84f9bb0122694f09b4215, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=731, earliestPutTs=1681314960965 2023-04-12 15:56:01,178 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] throttle.PressureAwareThroughputController(145): 4df381dfb8e2792d3aca65faf2673530#B#compaction#544 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:56:01,189 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4df381dfb8e2792d3aca65faf2673530#A#compaction#545 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:56:01,193 DEBUG [Thread-3125] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:56:01,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:56:01,195 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:56:01,203 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/B/6b437ef44e1444b5b3fbaae682e61e7f as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/6b437ef44e1444b5b3fbaae682e61e7f 2023-04-12 15:56:01,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:56:01,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:56:01,204 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:56:01,204 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:01,204 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:56:01,204 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:56:01,204 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:01,204 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:01,207 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 4df381dfb8e2792d3aca65faf2673530/B of 4df381dfb8e2792d3aca65faf2673530 into 6b437ef44e1444b5b3fbaae682e61e7f(size=13.5 K), total size for store is 13.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:56:01,207 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4df381dfb8e2792d3aca65faf2673530: 2023-04-12 15:56:01,207 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530., storeName=4df381dfb8e2792d3aca65faf2673530/B, priority=12, startTime=1681314961161; duration=0sec 2023-04-12 15:56:01,207 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:56:01,207 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:01,207 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-12 15:56:01,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:01,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:01,207 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:01,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:01,208 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:01,208 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:56:01,208 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:01,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:56:01,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:01,209 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:56:01,209 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 50242 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-12 15:56:01,209 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1912): 4df381dfb8e2792d3aca65faf2673530/C is initiating minor compaction (all files) 2023-04-12 15:56:01,209 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4df381dfb8e2792d3aca65faf2673530/C in TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:56:01,209 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/757c9ec80ddc4ff7b9732ccbabd4108d, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/3b96fea49487436a8e8d79d6de90772b, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/d17b9d8b01cc40aea7b330068885b9bc, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/83862bd5f02e4586ac96f31ac56d2000] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp, totalSize=49.1 K 2023-04-12 15:56:01,209 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 757c9ec80ddc4ff7b9732ccbabd4108d, keycount=150, bloomtype=ROW, size=13.4 K, encoding=NONE, compression=NONE, seqNum=646, earliestPutTs=1681314959033 2023-04-12 15:56:01,210 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 3b96fea49487436a8e8d79d6de90772b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=683, earliestPutTs=1681314959125 2023-04-12 15:56:01,210 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting d17b9d8b01cc40aea7b330068885b9bc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=695, earliestPutTs=1681314959772 2023-04-12 15:56:01,210 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 83862bd5f02e4586ac96f31ac56d2000, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=731, earliestPutTs=1681314960965 2023-04-12 15:56:01,212 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:56:01,212 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:56:01,214 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:56:01,214 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:01,214 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:56:01,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on 4df381dfb8e2792d3aca65faf2673530 2023-04-12 15:56:01,215 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4df381dfb8e2792d3aca65faf2673530 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-12 15:56:01,215 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=A 2023-04-12 15:56:01,215 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:01,215 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=B 2023-04-12 15:56:01,215 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:01,215 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=C 2023-04-12 15:56:01,215 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:01,221 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/A/fe3db05c466b4a64876ad55da57f6ba3 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/fe3db05c466b4a64876ad55da57f6ba3 2023-04-12 15:56:01,224 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:56:01,224 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:56:01,226 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 4df381dfb8e2792d3aca65faf2673530/A of 4df381dfb8e2792d3aca65faf2673530 into fe3db05c466b4a64876ad55da57f6ba3(size=13.5 K), total size for store is 13.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:56:01,226 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4df381dfb8e2792d3aca65faf2673530: 2023-04-12 15:56:01,226 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530., storeName=4df381dfb8e2792d3aca65faf2673530/A, priority=12, startTime=1681314961161; duration=0sec 2023-04-12 15:56:01,226 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:56:01,227 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] throttle.PressureAwareThroughputController(145): 4df381dfb8e2792d3aca65faf2673530#C#compaction#547 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:56:01,230 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:56:01,230 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. started... 2023-04-12 15:56:01,230 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. as already flushing 2023-04-12 15:56:01,230 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:56:01,230 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:56:01,230 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:56:01,230 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:01,230 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:56:01,230 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:56:01,232 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:01,232 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:01,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:01,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:01,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:01,232 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:01,232 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:56:01,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:01,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:01,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:01,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:01,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:01,233 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:56:01,233 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@755b2cd2[Count = 0] remaining members to acquire global barrier 2023-04-12 15:56:01,233 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:56:01,234 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:01,237 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:01,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:01,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:01,237 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:56:01,237 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:56:01,237 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:56:01,237 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:01,237 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:56:01,239 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:01,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:01,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:01,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:01,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:01,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:01,240 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:56:01,240 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:01,240 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:56:01,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:01,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:01,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:01,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:01,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:01,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:56:01,244 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:56:01,244 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:56:01,244 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:56:01,244 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:56:01,244 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:56:01,244 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:56:01,245 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:01,245 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:01,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:01,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:01,246 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:01,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:01,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:01,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:01,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:01,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:01,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:01,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:01,249 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:01,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:01,251 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:01,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:01,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:01,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:01,252 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/C/b263f34d63b843cbb927ed0a12c75c76 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/b263f34d63b843cbb927ed0a12c75c76 2023-04-12 15:56:01,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:01,253 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:01,253 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:01,253 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:01,253 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:01,253 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:01,253 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:01,253 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:01,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:01,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:01,253 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:56:01,254 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:56:01,254 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:01,254 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:01,253 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:01,254 DEBUG [Thread-3125] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:56:01,254 DEBUG [Thread-3125] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:56:01,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:01,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:01,254 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:56:01,256 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 4df381dfb8e2792d3aca65faf2673530/C of 4df381dfb8e2792d3aca65faf2673530 into b263f34d63b843cbb927ed0a12c75c76(size=13.5 K), total size for store is 13.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:56:01,256 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4df381dfb8e2792d3aca65faf2673530: 2023-04-12 15:56:01,256 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530., storeName=4df381dfb8e2792d3aca65faf2673530/C, priority=12, startTime=1681314961161; duration=0sec 2023-04-12 15:56:01,257 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:56:01,281 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:01,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 305 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55260 deadline: 1681315021258, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:01,282 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11854, totalHeapSize=27176, min timestamp=1681314961249, max timestamp=1681314961267Num uniques 859; 2023-04-12 15:56:01,282 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=11854, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-12 15:56:01,282 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11854, totalHeapSize=27176, min timestamp=1681314961249, max timestamp=1681314961267Num uniques 859; 2023-04-12 15:56:01,283 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=11854, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-12 15:56:01,283 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681314961251, max timestamp=1681314961267Num uniques 859; 2023-04-12 15:56:01,283 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-12 15:56:01,297 DEBUG [Thread-3126] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x78592df0 to 127.0.0.1:60154 2023-04-12 15:56:01,297 DEBUG [Thread-3123] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x426cb82a to 127.0.0.1:60154 2023-04-12 15:56:01,297 DEBUG [Thread-3126] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 15:56:01,297 DEBUG [Thread-3123] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 15:56:01,298 DEBUG [Thread-3128] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x659c68ab to 127.0.0.1:60154 2023-04-12 15:56:01,298 DEBUG [Thread-3128] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 15:56:01,299 DEBUG [Thread-3115] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x052203cb to 127.0.0.1:60154 2023-04-12 15:56:01,299 DEBUG [Thread-3115] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 15:56:01,302 DEBUG [Thread-3134] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4ac98310 to 127.0.0.1:60154 2023-04-12 15:56:01,302 DEBUG [Thread-3134] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 15:56:01,302 DEBUG [Thread-3119] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x51b8df2a to 127.0.0.1:60154 2023-04-12 15:56:01,302 DEBUG [Thread-3119] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 15:56:01,305 DEBUG [Thread-3130] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x680e773e to 127.0.0.1:60154 2023-04-12 15:56:01,305 DEBUG [Thread-3130] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 15:56:01,309 DEBUG [Thread-3132] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0b8f652e to 127.0.0.1:60154 2023-04-12 15:56:01,309 DEBUG [Thread-3132] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 15:56:01,354 DEBUG [Thread-3125] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:56:01,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:56:01,385 DEBUG [Thread-3121] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x67346c47 to 127.0.0.1:60154 2023-04-12 15:56:01,385 DEBUG [Thread-3121] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 15:56:01,637 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=743 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/A/beb69a13b3d04de295453427b4f6e390 2023-04-12 15:56:02,045 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=743 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/B/e6467c59e612490a87be158f4e8ddec8 2023-04-12 15:56:02,135 DEBUG [Thread-3117] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x420f945a to 127.0.0.1:60154 2023-04-12 15:56:02,136 DEBUG [Thread-3117] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 15:56:02,136 INFO [Listener at localhost/43583] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-12 15:56:02,136 INFO [Listener at localhost/43583] hbase.AcidGuaranteesTestTool(394): wrote 132 2023-04-12 15:56:02,136 INFO [Listener at localhost/43583] hbase.AcidGuaranteesTestTool(394): wrote 76 2023-04-12 15:56:02,136 INFO [Listener at localhost/43583] hbase.AcidGuaranteesTestTool(394): wrote 134 2023-04-12 15:56:02,136 INFO [Listener at localhost/43583] hbase.AcidGuaranteesTestTool(394): wrote 146 2023-04-12 15:56:02,136 INFO [Listener at localhost/43583] hbase.AcidGuaranteesTestTool(394): wrote 130 2023-04-12 15:56:02,136 INFO [Listener at localhost/43583] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-12 15:56:02,136 INFO [Listener at localhost/43583] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-12 15:56:02,136 INFO [Listener at localhost/43583] hbase.AcidGuaranteesTestTool(402): scanned 1866 2023-04-12 15:56:02,136 INFO [Listener at localhost/43583] hbase.AcidGuaranteesTestTool(403): verified 5598 rows 2023-04-12 15:56:02,136 INFO [Listener at localhost/43583] hbase.AcidGuaranteesTestTool(402): scanned 1860 2023-04-12 15:56:02,136 INFO [Listener at localhost/43583] hbase.AcidGuaranteesTestTool(403): verified 5580 rows 2023-04-12 15:56:02,136 INFO [Listener at localhost/43583] hbase.AcidGuaranteesTestTool(402): scanned 1854 2023-04-12 15:56:02,136 INFO [Listener at localhost/43583] hbase.AcidGuaranteesTestTool(403): verified 5562 rows 2023-04-12 15:56:02,136 INFO [Listener at localhost/43583] hbase.AcidGuaranteesTestTool(402): scanned 1870 2023-04-12 15:56:02,136 INFO [Listener at localhost/43583] hbase.AcidGuaranteesTestTool(403): verified 5607 rows 2023-04-12 15:56:02,136 INFO [Listener at localhost/43583] hbase.AcidGuaranteesTestTool(402): scanned 1870 2023-04-12 15:56:02,136 INFO [Listener at localhost/43583] hbase.AcidGuaranteesTestTool(403): verified 5610 rows 2023-04-12 15:56:02,136 INFO [Listener at localhost/43583] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-12 15:56:02,137 DEBUG [Listener at localhost/43583] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x15880ffb to 127.0.0.1:60154 2023-04-12 15:56:02,137 DEBUG [Listener at localhost/43583] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 15:56:02,141 INFO [Listener at localhost/43583] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-12 15:56:02,141 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-12 15:56:02,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure2.ProcedureExecutor(1029): Stored pid=50, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-12 15:56:02,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-12 15:56:02,145 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681314962145"}]},"ts":"1681314962145"} 2023-04-12 15:56:02,146 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-12 15:56:02,147 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-12 15:56:02,148 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=4df381dfb8e2792d3aca65faf2673530, UNASSIGN}] 2023-04-12 15:56:02,149 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=4df381dfb8e2792d3aca65faf2673530, UNASSIGN 2023-04-12 15:56:02,150 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=4df381dfb8e2792d3aca65faf2673530, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:02,150 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681314962150"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681314962150"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681314962150"}]},"ts":"1681314962150"} 2023-04-12 15:56:02,151 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=52, ppid=51, state=RUNNABLE; CloseRegionProcedure 4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739}] 2023-04-12 15:56:02,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-12 15:56:02,301 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 4df381dfb8e2792d3aca65faf2673530 2023-04-12 15:56:02,301 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 4df381dfb8e2792d3aca65faf2673530, disabling compactions & flushes 2023-04-12 15:56:02,301 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:56:02,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-12 15:56:02,453 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=743 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/C/9f0a6bb4bc28444f8007a60e87179dff 2023-04-12 15:56:02,457 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/A/beb69a13b3d04de295453427b4f6e390 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/beb69a13b3d04de295453427b4f6e390 2023-04-12 15:56:02,459 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/beb69a13b3d04de295453427b4f6e390, entries=200, sequenceid=743, filesize=14.3 K 2023-04-12 15:56:02,460 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/B/e6467c59e612490a87be158f4e8ddec8 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/e6467c59e612490a87be158f4e8ddec8 2023-04-12 15:56:02,463 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/e6467c59e612490a87be158f4e8ddec8, entries=150, sequenceid=743, filesize=11.9 K 2023-04-12 15:56:02,463 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/C/9f0a6bb4bc28444f8007a60e87179dff as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/9f0a6bb4bc28444f8007a60e87179dff 2023-04-12 15:56:02,466 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/9f0a6bb4bc28444f8007a60e87179dff, entries=150, sequenceid=743, filesize=11.9 K 2023-04-12 15:56:02,467 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=105.11 KB/107630 for 4df381dfb8e2792d3aca65faf2673530 in 1251ms, sequenceid=743, compaction requested=false 2023-04-12 15:56:02,467 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4df381dfb8e2792d3aca65faf2673530: 2023-04-12 15:56:02,467 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:56:02,467 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:56:02,467 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. after waiting 0 ms 2023-04-12 15:56:02,467 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:56:02,467 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 4df381dfb8e2792d3aca65faf2673530 3/3 column families, dataSize=105.11 KB heapSize=290.84 KB 2023-04-12 15:56:02,467 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=A 2023-04-12 15:56:02,467 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:56:02,467 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=B 2023-04-12 15:56:02,467 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:56:02,467 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4df381dfb8e2792d3aca65faf2673530, store=C 2023-04-12 15:56:02,467 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:56:02,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-12 15:56:02,873 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=776 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/A/8a91707acef84ad9832d5a383c49cee7 2023-04-12 15:56:03,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-12 15:56:03,284 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=776 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/B/83c90d44642549bdb4babb58bfae8692 2023-04-12 15:56:03,692 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=776 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/C/a14ec8c1afa34456b6458ea070ccbb86 2023-04-12 15:56:03,695 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/A/8a91707acef84ad9832d5a383c49cee7 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/8a91707acef84ad9832d5a383c49cee7 2023-04-12 15:56:03,698 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/8a91707acef84ad9832d5a383c49cee7, entries=150, sequenceid=776, filesize=11.9 K 2023-04-12 15:56:03,699 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/B/83c90d44642549bdb4babb58bfae8692 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/83c90d44642549bdb4babb58bfae8692 2023-04-12 15:56:03,703 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/83c90d44642549bdb4babb58bfae8692, entries=150, sequenceid=776, filesize=11.9 K 2023-04-12 15:56:03,703 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/.tmp/C/a14ec8c1afa34456b6458ea070ccbb86 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/a14ec8c1afa34456b6458ea070ccbb86 2023-04-12 15:56:03,706 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/a14ec8c1afa34456b6458ea070ccbb86, entries=150, sequenceid=776, filesize=11.9 K 2023-04-12 15:56:03,706 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~105.11 KB/107630, heapSize ~290.80 KB/297776, currentSize=0 B/0 for 4df381dfb8e2792d3aca65faf2673530 in 1239ms, sequenceid=776, compaction requested=true 2023-04-12 15:56:03,731 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/d3919cdd5ebc40be8714793670797f48, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/ea1d95daec7642c7afdd4fb099f40516, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/725b98bfe966453585d85de72d01c8ca, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/416d50d9887248049b766b97eb1716a0, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/9cd8ec3575044ec3863359f57eb1fe52, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/4f19c5585adc4da29df0032445408d3c, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/38c3d06ff4414f5aa00fdcaf10b496c9, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/01ef1847c2084f77a63accc1dfd5d0f7, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/a42b5fd96b5041358fe28898cc3ea568, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/ac2f5e8ce9a24bd1bdd0c04af1515f82, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/b01e45e87f434bf1bb0769639a1bbda6, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/826fceaef48449bf93a2341960586ea9, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/980d6114b8fe45428d3c143704798f6a, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/1ea3d3930ba44e338b3ba505142519ed, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/8f809a4c135e4c50b825f72bde5d7779, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/3839c82393bb4b51a9bc8caa97c84d5f, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/4c27ba2711f94a9e9343306b2bb30083, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/44d2d6670b8e4b7aaa2a23ac39753db2, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/7f74df07ee634e07a9d203b9293c351a, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/db153c3d9c604c89929693d2875d565d, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/200eb3c214764a0bbe41dbd340b59ea4, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/8da6814cb74e45b19fc42d659c2b1ff0, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/4591bbe1462d42afbdbf1beeff30a53b, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/efff4227ba284e4092a5e0a6980c92ee, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/79f0807411784c70a0a7432cff4a6fe8, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/b8c1479fd7674e339d71f972981bc1d4, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/3868f81de34644519cbd3dc64f55a7aa, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/d86e9d9a610a4366bf30c2ac9e64e0e0, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/99a9ad6cfbab4e659f38d9d9b9b3d270, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/f5a4cc23d1944422b0b06b4041c0a1e9, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/ec600f2c00ff478ba737ed5f2cc7479a, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/478f15888d08452ea36c03db8db2db59, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/c3b12e3a43b245b59c783ca2da5b739e, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/482a046ad0b74c6bb026a981aa09f873, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/b81c4df6f5e143c8a042716a465f844f, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/db64391f2fee420e916bff09b38f6d40, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/47d1715099b246619a48639dd7d3fd8b, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/90db606fc30447f2bac31ff8cad3541c, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/f02f5fcee75749cc88ccdece827a3005, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/5b4f058ea912496f93ca28483cd512a3, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/534031ffbd7e4e20a4804a5c05b41f7f, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/8a8619a4d75d45adb897bf20ea181a7e, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/ac125cec6e73480bb2f1c0a16875d395, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/78ea83de5dca46358001967218decd03, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/4521f4fd7e514ab092bac4e2c1747b13, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/ec2dc32db6494cfb8be5411acb56c307, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/adbe50515474414f803438c43deaeaae, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/e418f998054b4fd4b11787a4b8f91ed6, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/a0066a7d882e4d039d39401fb058db70] to archive 2023-04-12 15:56:03,732 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-12 15:56:03,733 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/d3919cdd5ebc40be8714793670797f48 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/d3919cdd5ebc40be8714793670797f48 2023-04-12 15:56:03,734 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/ea1d95daec7642c7afdd4fb099f40516 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/ea1d95daec7642c7afdd4fb099f40516 2023-04-12 15:56:03,735 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/725b98bfe966453585d85de72d01c8ca to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/725b98bfe966453585d85de72d01c8ca 2023-04-12 15:56:03,736 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/416d50d9887248049b766b97eb1716a0 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/416d50d9887248049b766b97eb1716a0 2023-04-12 15:56:03,736 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/9cd8ec3575044ec3863359f57eb1fe52 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/9cd8ec3575044ec3863359f57eb1fe52 2023-04-12 15:56:03,737 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/4f19c5585adc4da29df0032445408d3c to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/4f19c5585adc4da29df0032445408d3c 2023-04-12 15:56:03,738 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/38c3d06ff4414f5aa00fdcaf10b496c9 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/38c3d06ff4414f5aa00fdcaf10b496c9 2023-04-12 15:56:03,739 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/01ef1847c2084f77a63accc1dfd5d0f7 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/01ef1847c2084f77a63accc1dfd5d0f7 2023-04-12 15:56:03,740 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/a42b5fd96b5041358fe28898cc3ea568 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/a42b5fd96b5041358fe28898cc3ea568 2023-04-12 15:56:03,741 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/ac2f5e8ce9a24bd1bdd0c04af1515f82 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/ac2f5e8ce9a24bd1bdd0c04af1515f82 2023-04-12 15:56:03,742 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/b01e45e87f434bf1bb0769639a1bbda6 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/b01e45e87f434bf1bb0769639a1bbda6 2023-04-12 15:56:03,743 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/826fceaef48449bf93a2341960586ea9 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/826fceaef48449bf93a2341960586ea9 2023-04-12 15:56:03,744 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/980d6114b8fe45428d3c143704798f6a to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/980d6114b8fe45428d3c143704798f6a 2023-04-12 15:56:03,745 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/1ea3d3930ba44e338b3ba505142519ed to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/1ea3d3930ba44e338b3ba505142519ed 2023-04-12 15:56:03,745 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/8f809a4c135e4c50b825f72bde5d7779 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/8f809a4c135e4c50b825f72bde5d7779 2023-04-12 15:56:03,746 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/3839c82393bb4b51a9bc8caa97c84d5f to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/3839c82393bb4b51a9bc8caa97c84d5f 2023-04-12 15:56:03,747 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/4c27ba2711f94a9e9343306b2bb30083 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/4c27ba2711f94a9e9343306b2bb30083 2023-04-12 15:56:03,748 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/44d2d6670b8e4b7aaa2a23ac39753db2 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/44d2d6670b8e4b7aaa2a23ac39753db2 2023-04-12 15:56:03,749 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/7f74df07ee634e07a9d203b9293c351a to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/7f74df07ee634e07a9d203b9293c351a 2023-04-12 15:56:03,750 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/db153c3d9c604c89929693d2875d565d to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/db153c3d9c604c89929693d2875d565d 2023-04-12 15:56:03,751 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/200eb3c214764a0bbe41dbd340b59ea4 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/200eb3c214764a0bbe41dbd340b59ea4 2023-04-12 15:56:03,751 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/8da6814cb74e45b19fc42d659c2b1ff0 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/8da6814cb74e45b19fc42d659c2b1ff0 2023-04-12 15:56:03,752 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/4591bbe1462d42afbdbf1beeff30a53b to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/4591bbe1462d42afbdbf1beeff30a53b 2023-04-12 15:56:03,753 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/efff4227ba284e4092a5e0a6980c92ee to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/efff4227ba284e4092a5e0a6980c92ee 2023-04-12 15:56:03,754 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/79f0807411784c70a0a7432cff4a6fe8 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/79f0807411784c70a0a7432cff4a6fe8 2023-04-12 15:56:03,755 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/b8c1479fd7674e339d71f972981bc1d4 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/b8c1479fd7674e339d71f972981bc1d4 2023-04-12 15:56:03,756 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/3868f81de34644519cbd3dc64f55a7aa to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/3868f81de34644519cbd3dc64f55a7aa 2023-04-12 15:56:03,757 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/d86e9d9a610a4366bf30c2ac9e64e0e0 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/d86e9d9a610a4366bf30c2ac9e64e0e0 2023-04-12 15:56:03,757 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/99a9ad6cfbab4e659f38d9d9b9b3d270 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/99a9ad6cfbab4e659f38d9d9b9b3d270 2023-04-12 15:56:03,758 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/f5a4cc23d1944422b0b06b4041c0a1e9 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/f5a4cc23d1944422b0b06b4041c0a1e9 2023-04-12 15:56:03,759 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/ec600f2c00ff478ba737ed5f2cc7479a to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/ec600f2c00ff478ba737ed5f2cc7479a 2023-04-12 15:56:03,760 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/478f15888d08452ea36c03db8db2db59 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/478f15888d08452ea36c03db8db2db59 2023-04-12 15:56:03,761 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/c3b12e3a43b245b59c783ca2da5b739e to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/c3b12e3a43b245b59c783ca2da5b739e 2023-04-12 15:56:03,762 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/482a046ad0b74c6bb026a981aa09f873 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/482a046ad0b74c6bb026a981aa09f873 2023-04-12 15:56:03,762 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/b81c4df6f5e143c8a042716a465f844f to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/b81c4df6f5e143c8a042716a465f844f 2023-04-12 15:56:03,763 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/db64391f2fee420e916bff09b38f6d40 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/db64391f2fee420e916bff09b38f6d40 2023-04-12 15:56:03,764 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/47d1715099b246619a48639dd7d3fd8b to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/47d1715099b246619a48639dd7d3fd8b 2023-04-12 15:56:03,765 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/90db606fc30447f2bac31ff8cad3541c to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/90db606fc30447f2bac31ff8cad3541c 2023-04-12 15:56:03,766 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/f02f5fcee75749cc88ccdece827a3005 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/f02f5fcee75749cc88ccdece827a3005 2023-04-12 15:56:03,766 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/5b4f058ea912496f93ca28483cd512a3 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/5b4f058ea912496f93ca28483cd512a3 2023-04-12 15:56:03,767 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/534031ffbd7e4e20a4804a5c05b41f7f to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/534031ffbd7e4e20a4804a5c05b41f7f 2023-04-12 15:56:03,768 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/8a8619a4d75d45adb897bf20ea181a7e to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/8a8619a4d75d45adb897bf20ea181a7e 2023-04-12 15:56:03,769 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/ac125cec6e73480bb2f1c0a16875d395 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/ac125cec6e73480bb2f1c0a16875d395 2023-04-12 15:56:03,770 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/78ea83de5dca46358001967218decd03 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/78ea83de5dca46358001967218decd03 2023-04-12 15:56:03,770 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/4521f4fd7e514ab092bac4e2c1747b13 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/4521f4fd7e514ab092bac4e2c1747b13 2023-04-12 15:56:03,771 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/ec2dc32db6494cfb8be5411acb56c307 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/ec2dc32db6494cfb8be5411acb56c307 2023-04-12 15:56:03,772 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/adbe50515474414f803438c43deaeaae to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/adbe50515474414f803438c43deaeaae 2023-04-12 15:56:03,773 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/e418f998054b4fd4b11787a4b8f91ed6 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/e418f998054b4fd4b11787a4b8f91ed6 2023-04-12 15:56:03,773 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/a0066a7d882e4d039d39401fb058db70 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/a0066a7d882e4d039d39401fb058db70 2023-04-12 15:56:03,777 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/79eb74a3bda743dfb58a6637261f6e1a, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/055b65b5857a4098a881ede417a4cedf, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/7a421934a8794a71a2e53fc012068644, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/fed29ce026994db99cea3de2425667d3, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/b5f524c594bb48a6a9f9cd6aaa313d9d, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/f8d00bdf8dc34eb09fd76edfddbf4bb6, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/6a51ebeee7ef4460acf36a60e5e543ab, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/3dc36daffc874ce69e528b4bd10e9a69, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/3923040399a146cdbbe12bdac999a54c, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/1b2b9160212044558619e4e1f26dcd66, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/74e22206aba34b0196a6326ba5a9630b, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/870e6f9636cc4d209e233b8b7e62e3ab, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/75b5ec3a66644cdebbcc4ed8f0c811c9, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/be7919f2e0ea4dd9b6d7cbfe705a2f1b, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/6cd1c3b7e2924717bbeb45b7d634593f, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/e69ee74df29a4c06abfae7e87032f901, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/2ba63acda7614592825627ab3aa42793, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/245d539093b24f10a1a2fee1ebff2d12, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/6076d11358024647bddf2f498c26e762, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/2352bcafbea44bee821b6ac9592c3553, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/9179ee7e9ad24dafbfa8a9336dada40c, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/fae4637c71464ba093f316363f9bf3f1, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/f4eb93e542a64a2cbf0f750369b00147, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/f80962a0689242fba1a75853d3a7ddbc, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/a3147bec335e4006954fc4bbcb0defc1, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/482f0f11ea71403f846a451890f6cdeb, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/7dc6237b6f8a4a3199753193f1163ad3, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/14783d764e7c4fc38554d605ee98a205, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/61fec2329fdc4306ac0637dd065ea208, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/32e32955ee5d4841ab3bb471d0d971d5, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/035b5dfb907c47069b2ea091cf343ff0, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/078e1577ea1c42c380bc0c224966c6d1, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/9daf49a342ae46559cced96796e63846, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/cb82fa2cd7eb4d0ea39b88af5be5e899, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/041d95a3ca8743dfab3e58fe626137f4, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/ebde959533554bf2aa71ae822c08909b, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/fa12f6a5c0b74a30bcf6879275e79c91, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/44b25aaec44847d8a77f7c0214e88fea, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/2f6cffef02bd4ea7b8059df0fef34c3b, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/fe4dda7a758c42f8a1e79edbed7a0a00, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/ae707672897640efbc2509fc8c73cbaa, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/3888a786764040d9bd5f4f612353a0f8, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/eb545342586f4371ae4f99e266397212, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/c9227152399a45f89ea6c68ea178533d, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/00645d88478341d0828aa8ac26778a6b, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/ea3c1d04f38d41ff8ba7ef560e460d50, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/2ca974a5ed6647af8e1af7b263bc7815, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/b6c9129c55254f95bc20c3b5d8e5e584, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/f15120f788a84f9bb0122694f09b4215] to archive 2023-04-12 15:56:03,777 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-12 15:56:03,778 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/79eb74a3bda743dfb58a6637261f6e1a to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/79eb74a3bda743dfb58a6637261f6e1a 2023-04-12 15:56:03,779 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/055b65b5857a4098a881ede417a4cedf to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/055b65b5857a4098a881ede417a4cedf 2023-04-12 15:56:03,780 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/7a421934a8794a71a2e53fc012068644 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/7a421934a8794a71a2e53fc012068644 2023-04-12 15:56:03,781 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/fed29ce026994db99cea3de2425667d3 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/fed29ce026994db99cea3de2425667d3 2023-04-12 15:56:03,782 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/b5f524c594bb48a6a9f9cd6aaa313d9d to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/b5f524c594bb48a6a9f9cd6aaa313d9d 2023-04-12 15:56:03,782 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/f8d00bdf8dc34eb09fd76edfddbf4bb6 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/f8d00bdf8dc34eb09fd76edfddbf4bb6 2023-04-12 15:56:03,783 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/6a51ebeee7ef4460acf36a60e5e543ab to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/6a51ebeee7ef4460acf36a60e5e543ab 2023-04-12 15:56:03,784 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/3dc36daffc874ce69e528b4bd10e9a69 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/3dc36daffc874ce69e528b4bd10e9a69 2023-04-12 15:56:03,785 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/3923040399a146cdbbe12bdac999a54c to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/3923040399a146cdbbe12bdac999a54c 2023-04-12 15:56:03,785 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/1b2b9160212044558619e4e1f26dcd66 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/1b2b9160212044558619e4e1f26dcd66 2023-04-12 15:56:03,786 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/74e22206aba34b0196a6326ba5a9630b to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/74e22206aba34b0196a6326ba5a9630b 2023-04-12 15:56:03,787 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/870e6f9636cc4d209e233b8b7e62e3ab to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/870e6f9636cc4d209e233b8b7e62e3ab 2023-04-12 15:56:03,788 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/75b5ec3a66644cdebbcc4ed8f0c811c9 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/75b5ec3a66644cdebbcc4ed8f0c811c9 2023-04-12 15:56:03,788 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/be7919f2e0ea4dd9b6d7cbfe705a2f1b to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/be7919f2e0ea4dd9b6d7cbfe705a2f1b 2023-04-12 15:56:03,789 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/6cd1c3b7e2924717bbeb45b7d634593f to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/6cd1c3b7e2924717bbeb45b7d634593f 2023-04-12 15:56:03,790 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/e69ee74df29a4c06abfae7e87032f901 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/e69ee74df29a4c06abfae7e87032f901 2023-04-12 15:56:03,791 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/2ba63acda7614592825627ab3aa42793 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/2ba63acda7614592825627ab3aa42793 2023-04-12 15:56:03,792 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/245d539093b24f10a1a2fee1ebff2d12 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/245d539093b24f10a1a2fee1ebff2d12 2023-04-12 15:56:03,793 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/6076d11358024647bddf2f498c26e762 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/6076d11358024647bddf2f498c26e762 2023-04-12 15:56:03,793 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/2352bcafbea44bee821b6ac9592c3553 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/2352bcafbea44bee821b6ac9592c3553 2023-04-12 15:56:03,794 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/9179ee7e9ad24dafbfa8a9336dada40c to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/9179ee7e9ad24dafbfa8a9336dada40c 2023-04-12 15:56:03,795 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/fae4637c71464ba093f316363f9bf3f1 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/fae4637c71464ba093f316363f9bf3f1 2023-04-12 15:56:03,796 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/f4eb93e542a64a2cbf0f750369b00147 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/f4eb93e542a64a2cbf0f750369b00147 2023-04-12 15:56:03,797 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/f80962a0689242fba1a75853d3a7ddbc to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/f80962a0689242fba1a75853d3a7ddbc 2023-04-12 15:56:03,797 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/a3147bec335e4006954fc4bbcb0defc1 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/a3147bec335e4006954fc4bbcb0defc1 2023-04-12 15:56:03,798 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/482f0f11ea71403f846a451890f6cdeb to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/482f0f11ea71403f846a451890f6cdeb 2023-04-12 15:56:03,799 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/7dc6237b6f8a4a3199753193f1163ad3 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/7dc6237b6f8a4a3199753193f1163ad3 2023-04-12 15:56:03,800 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/14783d764e7c4fc38554d605ee98a205 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/14783d764e7c4fc38554d605ee98a205 2023-04-12 15:56:03,801 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/61fec2329fdc4306ac0637dd065ea208 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/61fec2329fdc4306ac0637dd065ea208 2023-04-12 15:56:03,801 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/32e32955ee5d4841ab3bb471d0d971d5 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/32e32955ee5d4841ab3bb471d0d971d5 2023-04-12 15:56:03,802 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/035b5dfb907c47069b2ea091cf343ff0 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/035b5dfb907c47069b2ea091cf343ff0 2023-04-12 15:56:03,803 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/078e1577ea1c42c380bc0c224966c6d1 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/078e1577ea1c42c380bc0c224966c6d1 2023-04-12 15:56:03,804 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/9daf49a342ae46559cced96796e63846 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/9daf49a342ae46559cced96796e63846 2023-04-12 15:56:03,804 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/cb82fa2cd7eb4d0ea39b88af5be5e899 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/cb82fa2cd7eb4d0ea39b88af5be5e899 2023-04-12 15:56:03,805 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/041d95a3ca8743dfab3e58fe626137f4 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/041d95a3ca8743dfab3e58fe626137f4 2023-04-12 15:56:03,806 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/ebde959533554bf2aa71ae822c08909b to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/ebde959533554bf2aa71ae822c08909b 2023-04-12 15:56:03,807 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/fa12f6a5c0b74a30bcf6879275e79c91 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/fa12f6a5c0b74a30bcf6879275e79c91 2023-04-12 15:56:03,808 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/44b25aaec44847d8a77f7c0214e88fea to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/44b25aaec44847d8a77f7c0214e88fea 2023-04-12 15:56:03,808 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/2f6cffef02bd4ea7b8059df0fef34c3b to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/2f6cffef02bd4ea7b8059df0fef34c3b 2023-04-12 15:56:03,809 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/fe4dda7a758c42f8a1e79edbed7a0a00 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/fe4dda7a758c42f8a1e79edbed7a0a00 2023-04-12 15:56:03,810 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/ae707672897640efbc2509fc8c73cbaa to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/ae707672897640efbc2509fc8c73cbaa 2023-04-12 15:56:03,811 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/3888a786764040d9bd5f4f612353a0f8 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/3888a786764040d9bd5f4f612353a0f8 2023-04-12 15:56:03,812 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/eb545342586f4371ae4f99e266397212 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/eb545342586f4371ae4f99e266397212 2023-04-12 15:56:03,812 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/c9227152399a45f89ea6c68ea178533d to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/c9227152399a45f89ea6c68ea178533d 2023-04-12 15:56:03,813 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/00645d88478341d0828aa8ac26778a6b to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/00645d88478341d0828aa8ac26778a6b 2023-04-12 15:56:03,814 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/ea3c1d04f38d41ff8ba7ef560e460d50 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/ea3c1d04f38d41ff8ba7ef560e460d50 2023-04-12 15:56:03,815 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/2ca974a5ed6647af8e1af7b263bc7815 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/2ca974a5ed6647af8e1af7b263bc7815 2023-04-12 15:56:03,816 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/b6c9129c55254f95bc20c3b5d8e5e584 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/b6c9129c55254f95bc20c3b5d8e5e584 2023-04-12 15:56:03,817 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/f15120f788a84f9bb0122694f09b4215 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/f15120f788a84f9bb0122694f09b4215 2023-04-12 15:56:03,820 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/7c0ede0be0e64184a182cb2c16de4796, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/cd6a46adcece4549aa04f48ee04f698f, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/5502d99bac8e4426b6b3df563d9c86a2, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/4993656bbde34d1096a4c893bb940815, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/5ee1d0c731b94394bc32f0822486bdc8, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/327a0142253340ffa28b0bcefb7a06dd, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/0953ddd414e94905b98b32b88c8091fc, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/6a2dbc477d024c35ba743529d4cb5e3d, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/6065b13feb0c461db9e3bf535ef2c109, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/5b7ec464d1624b718927e56e7170276e, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/0b8b3872afca4544b5a97ec8f8af02c0, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/415c0b98489944ebb2e83e4e1e67bc33, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/787c417108be47179d30b0314a119c6d, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/31a21a91e34f4f60b6d0c1b45f7f00e6, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/35c59df13bdb406d80baefe235282e4f, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/58d42b1c4659480ba53d994a27c6290e, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/ec2830ae4b494b0baeeccb2cdf5a48f2, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/cc899cecff9c4628bf33e7a71989a13c, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/5bffd9da81d64cf88e694fdcca264c95, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/7a0a6ac02e964c49aea85aed5775da4d, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/18a9bb7e91414ccca206b47d2e80cd78, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/a7bd8fb4177d4b46bcbb0c2c843c5802, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/a685dbb2493249d997efedbed80d4b7e, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/acceaec84a894f129dd3c752e71a28c9, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/3604312e5f944485bace7fd7977a9c96, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/19902c3a096a4c17857583079aeb6ee4, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/45e685c74f0948ac80026e409c72a3fe, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/6e7333c2a545494d8452781dab218e1f, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/6303bac5909042efa692f06a2cbef356, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/d02c7ea22ce34295b782128619720dae, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/aa85961bdecb4c63bf0d35e6d047608a, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/06b0d58b1a7b4e6c9e199aeada295bae, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/84e1281e7fc2430c86e3531719541b35, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/80821659e1dd46e5991805b1a0bd4220, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/3408ae57524c4728bd7330af5cd723f8, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/f6104a9563bf4827bfa9bbaadd207566, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/a491231d287c492e888737e4602c64e9, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/7c31f63dc8ec488aa81f08ba1d15feb2, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/adc2d7bba1154a52889909aa0a14cca5, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/da5112797d2a45db884098f2c1b637f9, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/3e7cc01fb1b743738a5579039c4940d3, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/303ae31417c0418fb2f04e4b817913d7, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/e131c346d6bd460b9abdc588605fb936, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/757c9ec80ddc4ff7b9732ccbabd4108d, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/3da6a04c762c4994be6fb21b2c57aa4c, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/3b96fea49487436a8e8d79d6de90772b, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/d17b9d8b01cc40aea7b330068885b9bc, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/83862bd5f02e4586ac96f31ac56d2000] to archive 2023-04-12 15:56:03,820 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-12 15:56:03,821 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/7c0ede0be0e64184a182cb2c16de4796 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/7c0ede0be0e64184a182cb2c16de4796 2023-04-12 15:56:03,822 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/cd6a46adcece4549aa04f48ee04f698f to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/cd6a46adcece4549aa04f48ee04f698f 2023-04-12 15:56:03,823 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/5502d99bac8e4426b6b3df563d9c86a2 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/5502d99bac8e4426b6b3df563d9c86a2 2023-04-12 15:56:03,824 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/4993656bbde34d1096a4c893bb940815 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/4993656bbde34d1096a4c893bb940815 2023-04-12 15:56:03,824 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/5ee1d0c731b94394bc32f0822486bdc8 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/5ee1d0c731b94394bc32f0822486bdc8 2023-04-12 15:56:03,825 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/327a0142253340ffa28b0bcefb7a06dd to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/327a0142253340ffa28b0bcefb7a06dd 2023-04-12 15:56:03,826 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/0953ddd414e94905b98b32b88c8091fc to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/0953ddd414e94905b98b32b88c8091fc 2023-04-12 15:56:03,827 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/6a2dbc477d024c35ba743529d4cb5e3d to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/6a2dbc477d024c35ba743529d4cb5e3d 2023-04-12 15:56:03,827 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/6065b13feb0c461db9e3bf535ef2c109 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/6065b13feb0c461db9e3bf535ef2c109 2023-04-12 15:56:03,828 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/5b7ec464d1624b718927e56e7170276e to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/5b7ec464d1624b718927e56e7170276e 2023-04-12 15:56:03,829 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/0b8b3872afca4544b5a97ec8f8af02c0 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/0b8b3872afca4544b5a97ec8f8af02c0 2023-04-12 15:56:03,830 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/415c0b98489944ebb2e83e4e1e67bc33 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/415c0b98489944ebb2e83e4e1e67bc33 2023-04-12 15:56:03,830 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/787c417108be47179d30b0314a119c6d to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/787c417108be47179d30b0314a119c6d 2023-04-12 15:56:03,831 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/31a21a91e34f4f60b6d0c1b45f7f00e6 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/31a21a91e34f4f60b6d0c1b45f7f00e6 2023-04-12 15:56:03,832 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/35c59df13bdb406d80baefe235282e4f to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/35c59df13bdb406d80baefe235282e4f 2023-04-12 15:56:03,833 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/58d42b1c4659480ba53d994a27c6290e to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/58d42b1c4659480ba53d994a27c6290e 2023-04-12 15:56:03,833 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/ec2830ae4b494b0baeeccb2cdf5a48f2 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/ec2830ae4b494b0baeeccb2cdf5a48f2 2023-04-12 15:56:03,834 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/cc899cecff9c4628bf33e7a71989a13c to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/cc899cecff9c4628bf33e7a71989a13c 2023-04-12 15:56:03,835 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/5bffd9da81d64cf88e694fdcca264c95 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/5bffd9da81d64cf88e694fdcca264c95 2023-04-12 15:56:03,836 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/7a0a6ac02e964c49aea85aed5775da4d to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/7a0a6ac02e964c49aea85aed5775da4d 2023-04-12 15:56:03,836 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/18a9bb7e91414ccca206b47d2e80cd78 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/18a9bb7e91414ccca206b47d2e80cd78 2023-04-12 15:56:03,837 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/a7bd8fb4177d4b46bcbb0c2c843c5802 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/a7bd8fb4177d4b46bcbb0c2c843c5802 2023-04-12 15:56:03,838 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/a685dbb2493249d997efedbed80d4b7e to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/a685dbb2493249d997efedbed80d4b7e 2023-04-12 15:56:03,839 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/acceaec84a894f129dd3c752e71a28c9 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/acceaec84a894f129dd3c752e71a28c9 2023-04-12 15:56:03,840 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/3604312e5f944485bace7fd7977a9c96 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/3604312e5f944485bace7fd7977a9c96 2023-04-12 15:56:03,840 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/19902c3a096a4c17857583079aeb6ee4 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/19902c3a096a4c17857583079aeb6ee4 2023-04-12 15:56:03,841 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/45e685c74f0948ac80026e409c72a3fe to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/45e685c74f0948ac80026e409c72a3fe 2023-04-12 15:56:03,842 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/6e7333c2a545494d8452781dab218e1f to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/6e7333c2a545494d8452781dab218e1f 2023-04-12 15:56:03,843 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/6303bac5909042efa692f06a2cbef356 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/6303bac5909042efa692f06a2cbef356 2023-04-12 15:56:03,844 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/d02c7ea22ce34295b782128619720dae to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/d02c7ea22ce34295b782128619720dae 2023-04-12 15:56:03,844 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/aa85961bdecb4c63bf0d35e6d047608a to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/aa85961bdecb4c63bf0d35e6d047608a 2023-04-12 15:56:03,845 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/06b0d58b1a7b4e6c9e199aeada295bae to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/06b0d58b1a7b4e6c9e199aeada295bae 2023-04-12 15:56:03,846 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/84e1281e7fc2430c86e3531719541b35 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/84e1281e7fc2430c86e3531719541b35 2023-04-12 15:56:03,847 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/80821659e1dd46e5991805b1a0bd4220 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/80821659e1dd46e5991805b1a0bd4220 2023-04-12 15:56:03,848 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/3408ae57524c4728bd7330af5cd723f8 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/3408ae57524c4728bd7330af5cd723f8 2023-04-12 15:56:03,849 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/f6104a9563bf4827bfa9bbaadd207566 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/f6104a9563bf4827bfa9bbaadd207566 2023-04-12 15:56:03,850 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/a491231d287c492e888737e4602c64e9 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/a491231d287c492e888737e4602c64e9 2023-04-12 15:56:03,851 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/7c31f63dc8ec488aa81f08ba1d15feb2 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/7c31f63dc8ec488aa81f08ba1d15feb2 2023-04-12 15:56:03,852 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/adc2d7bba1154a52889909aa0a14cca5 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/adc2d7bba1154a52889909aa0a14cca5 2023-04-12 15:56:03,852 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/da5112797d2a45db884098f2c1b637f9 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/da5112797d2a45db884098f2c1b637f9 2023-04-12 15:56:03,853 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/3e7cc01fb1b743738a5579039c4940d3 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/3e7cc01fb1b743738a5579039c4940d3 2023-04-12 15:56:03,854 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/303ae31417c0418fb2f04e4b817913d7 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/303ae31417c0418fb2f04e4b817913d7 2023-04-12 15:56:03,855 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/e131c346d6bd460b9abdc588605fb936 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/e131c346d6bd460b9abdc588605fb936 2023-04-12 15:56:03,856 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/757c9ec80ddc4ff7b9732ccbabd4108d to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/757c9ec80ddc4ff7b9732ccbabd4108d 2023-04-12 15:56:03,857 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/3da6a04c762c4994be6fb21b2c57aa4c to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/3da6a04c762c4994be6fb21b2c57aa4c 2023-04-12 15:56:03,857 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/3b96fea49487436a8e8d79d6de90772b to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/3b96fea49487436a8e8d79d6de90772b 2023-04-12 15:56:03,858 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/d17b9d8b01cc40aea7b330068885b9bc to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/d17b9d8b01cc40aea7b330068885b9bc 2023-04-12 15:56:03,859 DEBUG [StoreCloser-TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/83862bd5f02e4586ac96f31ac56d2000 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/83862bd5f02e4586ac96f31ac56d2000 2023-04-12 15:56:03,863 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/recovered.edits/779.seqid, newMaxSeqId=779, maxSeqId=1 2023-04-12 15:56:03,864 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530. 2023-04-12 15:56:03,864 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 4df381dfb8e2792d3aca65faf2673530: 2023-04-12 15:56:03,865 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 4df381dfb8e2792d3aca65faf2673530 2023-04-12 15:56:03,865 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=4df381dfb8e2792d3aca65faf2673530, regionState=CLOSED 2023-04-12 15:56:03,866 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681314963865"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681314963865"}]},"ts":"1681314963865"} 2023-04-12 15:56:03,868 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=52, resume processing ppid=51 2023-04-12 15:56:03,868 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=52, ppid=51, state=SUCCESS; CloseRegionProcedure 4df381dfb8e2792d3aca65faf2673530, server=jenkins-hbase4.apache.org,38905,1681314818739 in 1.7160 sec 2023-04-12 15:56:03,869 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=51, resume processing ppid=50 2023-04-12 15:56:03,869 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=51, ppid=50, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=4df381dfb8e2792d3aca65faf2673530, UNASSIGN in 1.7200 sec 2023-04-12 15:56:03,869 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681314963869"}]},"ts":"1681314963869"} 2023-04-12 15:56:03,870 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-12 15:56:03,871 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-12 15:56:03,872 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=50, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.7310 sec 2023-04-12 15:56:04,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-12 15:56:04,249 INFO [Listener at localhost/43583] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 50 completed 2023-04-12 15:56:04,250 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-12 15:56:04,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure2.ProcedureExecutor(1029): Stored pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-12 15:56:04,252 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-12 15:56:04,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-04-12 15:56:04,253 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=53, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-12 15:56:04,257 DEBUG [HFileArchiver-11] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530 2023-04-12 15:56:04,259 DEBUG [HFileArchiver-11] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A, FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B, FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C, FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/recovered.edits] 2023-04-12 15:56:04,261 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/8a91707acef84ad9832d5a383c49cee7 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/8a91707acef84ad9832d5a383c49cee7 2023-04-12 15:56:04,262 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/beb69a13b3d04de295453427b4f6e390 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/beb69a13b3d04de295453427b4f6e390 2023-04-12 15:56:04,263 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/fe3db05c466b4a64876ad55da57f6ba3 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/A/fe3db05c466b4a64876ad55da57f6ba3 2023-04-12 15:56:04,265 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/6b437ef44e1444b5b3fbaae682e61e7f to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/6b437ef44e1444b5b3fbaae682e61e7f 2023-04-12 15:56:04,266 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/83c90d44642549bdb4babb58bfae8692 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/83c90d44642549bdb4babb58bfae8692 2023-04-12 15:56:04,267 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/e6467c59e612490a87be158f4e8ddec8 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/B/e6467c59e612490a87be158f4e8ddec8 2023-04-12 15:56:04,269 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/9f0a6bb4bc28444f8007a60e87179dff to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/9f0a6bb4bc28444f8007a60e87179dff 2023-04-12 15:56:04,270 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/a14ec8c1afa34456b6458ea070ccbb86 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/a14ec8c1afa34456b6458ea070ccbb86 2023-04-12 15:56:04,270 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/b263f34d63b843cbb927ed0a12c75c76 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/C/b263f34d63b843cbb927ed0a12c75c76 2023-04-12 15:56:04,272 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/recovered.edits/779.seqid to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530/recovered.edits/779.seqid 2023-04-12 15:56:04,273 DEBUG [HFileArchiver-11] backup.HFileArchiver(596): Deleted hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/4df381dfb8e2792d3aca65faf2673530 2023-04-12 15:56:04,273 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-12 15:56:04,274 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=53, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-12 15:56:04,275 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-12 15:56:04,277 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-12 15:56:04,278 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=53, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-12 15:56:04,278 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-12 15:56:04,278 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681314964278"}]},"ts":"9223372036854775807"} 2023-04-12 15:56:04,279 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-12 15:56:04,279 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 4df381dfb8e2792d3aca65faf2673530, NAME => 'TestAcidGuarantees,,1681314940085.4df381dfb8e2792d3aca65faf2673530.', STARTKEY => '', ENDKEY => ''}] 2023-04-12 15:56:04,279 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-12 15:56:04,279 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681314964279"}]},"ts":"9223372036854775807"} 2023-04-12 15:56:04,280 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-12 15:56:04,283 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=53, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-12 15:56:04,284 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=53, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 33 msec 2023-04-12 15:56:04,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-04-12 15:56:04,354 INFO [Listener at localhost/43583] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 53 completed 2023-04-12 15:56:04,365 INFO [Listener at localhost/43583] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=232 (was 229) - Thread LEAK? -, OpenFileDescriptor=576 (was 545) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=396 (was 345) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=860 (was 1259) 2023-04-12 15:56:04,378 INFO [Listener at localhost/43583] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=232, OpenFileDescriptor=576, MaxFileDescriptor=60000, SystemLoadAverage=396, ProcessCount=169, AvailableMemoryMB=860 2023-04-12 15:56:04,379 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-12 15:56:04,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-12 15:56:04,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure2.ProcedureExecutor(1029): Stored pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-12 15:56:04,382 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-12 15:56:04,382 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 54 2023-04-12 15:56:04,382 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-12 15:56:04,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-12 15:56:04,383 DEBUG [HFileArchiver-12] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03 2023-04-12 15:56:04,384 DEBUG [HFileArchiver-12] backup.HFileArchiver(153): Directory hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03 empty. 2023-04-12 15:56:04,384 DEBUG [HFileArchiver-12] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03 2023-04-12 15:56:04,384 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-12 15:56:04,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-12 15:56:04,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-12 15:56:04,791 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-12 15:56:04,792 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 4870003209f366bc50ef2084e6e2cf03, NAME => 'TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp 2023-04-12 15:56:04,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-12 15:56:05,198 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-12 15:56:05,198 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 4870003209f366bc50ef2084e6e2cf03, disabling compactions & flushes 2023-04-12 15:56:05,198 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:05,198 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:05,198 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. after waiting 0 ms 2023-04-12 15:56:05,198 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:05,198 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:05,198 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 4870003209f366bc50ef2084e6e2cf03: 2023-04-12 15:56:05,200 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-12 15:56:05,201 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681314965201"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681314965201"}]},"ts":"1681314965201"} 2023-04-12 15:56:05,202 INFO [PEWorker-4] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-12 15:56:05,202 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-12 15:56:05,202 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681314965202"}]},"ts":"1681314965202"} 2023-04-12 15:56:05,203 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-12 15:56:05,207 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=4870003209f366bc50ef2084e6e2cf03, ASSIGN}] 2023-04-12 15:56:05,208 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=4870003209f366bc50ef2084e6e2cf03, ASSIGN 2023-04-12 15:56:05,209 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=4870003209f366bc50ef2084e6e2cf03, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,38905,1681314818739; forceNewPlan=false, retain=false 2023-04-12 15:56:05,360 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=4870003209f366bc50ef2084e6e2cf03, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:05,360 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681314965360"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681314965360"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681314965360"}]},"ts":"1681314965360"} 2023-04-12 15:56:05,361 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=56, ppid=55, state=RUNNABLE; OpenRegionProcedure 4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739}] 2023-04-12 15:56:05,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-12 15:56:05,516 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:05,516 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 4870003209f366bc50ef2084e6e2cf03, NAME => 'TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.', STARTKEY => '', ENDKEY => ''} 2023-04-12 15:56:05,516 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 4870003209f366bc50ef2084e6e2cf03 2023-04-12 15:56:05,516 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-12 15:56:05,516 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 4870003209f366bc50ef2084e6e2cf03 2023-04-12 15:56:05,516 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 4870003209f366bc50ef2084e6e2cf03 2023-04-12 15:56:05,517 INFO [StoreOpener-4870003209f366bc50ef2084e6e2cf03-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 4870003209f366bc50ef2084e6e2cf03 2023-04-12 15:56:05,518 DEBUG [StoreOpener-4870003209f366bc50ef2084e6e2cf03-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A 2023-04-12 15:56:05,518 DEBUG [StoreOpener-4870003209f366bc50ef2084e6e2cf03-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A 2023-04-12 15:56:05,518 INFO [StoreOpener-4870003209f366bc50ef2084e6e2cf03-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-12 15:56:05,519 INFO [StoreOpener-4870003209f366bc50ef2084e6e2cf03-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4870003209f366bc50ef2084e6e2cf03 columnFamilyName A 2023-04-12 15:56:05,519 INFO [StoreOpener-4870003209f366bc50ef2084e6e2cf03-1] regionserver.HStore(310): Store=4870003209f366bc50ef2084e6e2cf03/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 15:56:05,519 INFO [StoreOpener-4870003209f366bc50ef2084e6e2cf03-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 4870003209f366bc50ef2084e6e2cf03 2023-04-12 15:56:05,520 DEBUG [StoreOpener-4870003209f366bc50ef2084e6e2cf03-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B 2023-04-12 15:56:05,520 DEBUG [StoreOpener-4870003209f366bc50ef2084e6e2cf03-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B 2023-04-12 15:56:05,520 INFO [StoreOpener-4870003209f366bc50ef2084e6e2cf03-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-12 15:56:05,520 INFO [StoreOpener-4870003209f366bc50ef2084e6e2cf03-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4870003209f366bc50ef2084e6e2cf03 columnFamilyName B 2023-04-12 15:56:05,521 INFO [StoreOpener-4870003209f366bc50ef2084e6e2cf03-1] regionserver.HStore(310): Store=4870003209f366bc50ef2084e6e2cf03/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 15:56:05,521 INFO [StoreOpener-4870003209f366bc50ef2084e6e2cf03-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 4870003209f366bc50ef2084e6e2cf03 2023-04-12 15:56:05,522 DEBUG [StoreOpener-4870003209f366bc50ef2084e6e2cf03-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C 2023-04-12 15:56:05,522 DEBUG [StoreOpener-4870003209f366bc50ef2084e6e2cf03-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C 2023-04-12 15:56:05,522 INFO [StoreOpener-4870003209f366bc50ef2084e6e2cf03-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-12 15:56:05,522 INFO [StoreOpener-4870003209f366bc50ef2084e6e2cf03-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4870003209f366bc50ef2084e6e2cf03 columnFamilyName C 2023-04-12 15:56:05,523 INFO [StoreOpener-4870003209f366bc50ef2084e6e2cf03-1] regionserver.HStore(310): Store=4870003209f366bc50ef2084e6e2cf03/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 15:56:05,523 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:05,523 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03 2023-04-12 15:56:05,524 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03 2023-04-12 15:56:05,526 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-12 15:56:05,527 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 4870003209f366bc50ef2084e6e2cf03 2023-04-12 15:56:05,528 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-12 15:56:05,528 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 4870003209f366bc50ef2084e6e2cf03; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61097537, jitterRate=-0.08957575261592865}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-12 15:56:05,528 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 4870003209f366bc50ef2084e6e2cf03: 2023-04-12 15:56:05,529 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03., pid=56, masterSystemTime=1681314965513 2023-04-12 15:56:05,530 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:05,530 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:05,531 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=4870003209f366bc50ef2084e6e2cf03, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:05,531 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681314965531"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681314965531"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681314965531"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681314965531"}]},"ts":"1681314965531"} 2023-04-12 15:56:05,533 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=56, resume processing ppid=55 2023-04-12 15:56:05,533 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=56, ppid=55, state=SUCCESS; OpenRegionProcedure 4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 in 171 msec 2023-04-12 15:56:05,534 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=55, resume processing ppid=54 2023-04-12 15:56:05,534 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=55, ppid=54, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=4870003209f366bc50ef2084e6e2cf03, ASSIGN in 326 msec 2023-04-12 15:56:05,534 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-12 15:56:05,534 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681314965534"}]},"ts":"1681314965534"} 2023-04-12 15:56:05,535 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-12 15:56:05,537 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-12 15:56:05,538 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=54, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1570 sec 2023-04-12 15:56:06,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-12 15:56:06,488 INFO [Listener at localhost/43583] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 54 completed 2023-04-12 15:56:06,489 DEBUG [Listener at localhost/43583] zookeeper.ReadOnlyZKClient(139): Connect 0x5f9e8654 to 127.0.0.1:60154 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 15:56:06,494 DEBUG [Listener at localhost/43583] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@62b3caa, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 15:56:06,495 DEBUG [Listener at localhost/43583] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 15:56:06,496 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53278, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 15:56:06,497 DEBUG [Listener at localhost/43583] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-12 15:56:06,499 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54332, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-12 15:56:06,500 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-12 15:56:06,501 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-12 15:56:06,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure2.ProcedureExecutor(1029): Stored pid=57, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-12 15:56:06,914 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-12 15:56:06,915 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-12 15:56:06,915 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-12 15:56:06,916 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=58, ppid=57, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-12 15:56:06,919 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=4870003209f366bc50ef2084e6e2cf03, REOPEN/MOVE}] 2023-04-12 15:56:06,919 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=4870003209f366bc50ef2084e6e2cf03, REOPEN/MOVE 2023-04-12 15:56:06,920 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=4870003209f366bc50ef2084e6e2cf03, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:06,920 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681314966920"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681314966920"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681314966920"}]},"ts":"1681314966920"} 2023-04-12 15:56:06,921 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=60, ppid=59, state=RUNNABLE; CloseRegionProcedure 4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739}] 2023-04-12 15:56:07,073 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 4870003209f366bc50ef2084e6e2cf03 2023-04-12 15:56:07,074 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 4870003209f366bc50ef2084e6e2cf03, disabling compactions & flushes 2023-04-12 15:56:07,074 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:07,074 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:07,074 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. after waiting 0 ms 2023-04-12 15:56:07,074 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:07,077 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-12 15:56:07,078 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:07,078 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 4870003209f366bc50ef2084e6e2cf03: 2023-04-12 15:56:07,078 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 4870003209f366bc50ef2084e6e2cf03 to self. 2023-04-12 15:56:07,079 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 4870003209f366bc50ef2084e6e2cf03 2023-04-12 15:56:07,079 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=4870003209f366bc50ef2084e6e2cf03, regionState=CLOSED 2023-04-12 15:56:07,079 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681314967079"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681314967079"}]},"ts":"1681314967079"} 2023-04-12 15:56:07,081 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=60, resume processing ppid=59 2023-04-12 15:56:07,081 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=60, ppid=59, state=SUCCESS; CloseRegionProcedure 4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 in 159 msec 2023-04-12 15:56:07,081 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=4870003209f366bc50ef2084e6e2cf03, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,38905,1681314818739; forceNewPlan=false, retain=true 2023-04-12 15:56:07,232 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=4870003209f366bc50ef2084e6e2cf03, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:07,232 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681314967232"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681314967232"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681314967232"}]},"ts":"1681314967232"} 2023-04-12 15:56:07,233 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=61, ppid=59, state=RUNNABLE; OpenRegionProcedure 4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739}] 2023-04-12 15:56:07,388 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:07,388 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 4870003209f366bc50ef2084e6e2cf03, NAME => 'TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.', STARTKEY => '', ENDKEY => ''} 2023-04-12 15:56:07,388 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 4870003209f366bc50ef2084e6e2cf03 2023-04-12 15:56:07,388 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-12 15:56:07,388 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 4870003209f366bc50ef2084e6e2cf03 2023-04-12 15:56:07,388 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 4870003209f366bc50ef2084e6e2cf03 2023-04-12 15:56:07,389 INFO [StoreOpener-4870003209f366bc50ef2084e6e2cf03-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 4870003209f366bc50ef2084e6e2cf03 2023-04-12 15:56:07,390 DEBUG [StoreOpener-4870003209f366bc50ef2084e6e2cf03-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A 2023-04-12 15:56:07,390 DEBUG [StoreOpener-4870003209f366bc50ef2084e6e2cf03-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A 2023-04-12 15:56:07,391 INFO [StoreOpener-4870003209f366bc50ef2084e6e2cf03-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-12 15:56:07,391 INFO [StoreOpener-4870003209f366bc50ef2084e6e2cf03-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4870003209f366bc50ef2084e6e2cf03 columnFamilyName A 2023-04-12 15:56:07,392 INFO [StoreOpener-4870003209f366bc50ef2084e6e2cf03-1] regionserver.HStore(310): Store=4870003209f366bc50ef2084e6e2cf03/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 15:56:07,392 INFO [StoreOpener-4870003209f366bc50ef2084e6e2cf03-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 4870003209f366bc50ef2084e6e2cf03 2023-04-12 15:56:07,393 DEBUG [StoreOpener-4870003209f366bc50ef2084e6e2cf03-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B 2023-04-12 15:56:07,393 DEBUG [StoreOpener-4870003209f366bc50ef2084e6e2cf03-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B 2023-04-12 15:56:07,393 INFO [StoreOpener-4870003209f366bc50ef2084e6e2cf03-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-12 15:56:07,393 INFO [StoreOpener-4870003209f366bc50ef2084e6e2cf03-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4870003209f366bc50ef2084e6e2cf03 columnFamilyName B 2023-04-12 15:56:07,394 INFO [StoreOpener-4870003209f366bc50ef2084e6e2cf03-1] regionserver.HStore(310): Store=4870003209f366bc50ef2084e6e2cf03/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 15:56:07,394 INFO [StoreOpener-4870003209f366bc50ef2084e6e2cf03-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 4870003209f366bc50ef2084e6e2cf03 2023-04-12 15:56:07,394 DEBUG [StoreOpener-4870003209f366bc50ef2084e6e2cf03-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C 2023-04-12 15:56:07,394 DEBUG [StoreOpener-4870003209f366bc50ef2084e6e2cf03-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C 2023-04-12 15:56:07,395 INFO [StoreOpener-4870003209f366bc50ef2084e6e2cf03-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-12 15:56:07,395 INFO [StoreOpener-4870003209f366bc50ef2084e6e2cf03-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4870003209f366bc50ef2084e6e2cf03 columnFamilyName C 2023-04-12 15:56:07,395 INFO [StoreOpener-4870003209f366bc50ef2084e6e2cf03-1] regionserver.HStore(310): Store=4870003209f366bc50ef2084e6e2cf03/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 15:56:07,395 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:07,396 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03 2023-04-12 15:56:07,397 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03 2023-04-12 15:56:07,398 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-12 15:56:07,399 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 4870003209f366bc50ef2084e6e2cf03 2023-04-12 15:56:07,400 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 4870003209f366bc50ef2084e6e2cf03; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68345641, jitterRate=0.018429413437843323}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-12 15:56:07,400 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 4870003209f366bc50ef2084e6e2cf03: 2023-04-12 15:56:07,401 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03., pid=61, masterSystemTime=1681314967384 2023-04-12 15:56:07,402 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:07,402 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:07,402 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=4870003209f366bc50ef2084e6e2cf03, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:07,402 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681314967402"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681314967402"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681314967402"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681314967402"}]},"ts":"1681314967402"} 2023-04-12 15:56:07,404 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=61, resume processing ppid=59 2023-04-12 15:56:07,404 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=61, ppid=59, state=SUCCESS; OpenRegionProcedure 4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 in 170 msec 2023-04-12 15:56:07,405 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=59, resume processing ppid=58 2023-04-12 15:56:07,405 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=59, ppid=58, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=4870003209f366bc50ef2084e6e2cf03, REOPEN/MOVE in 485 msec 2023-04-12 15:56:07,407 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=58, resume processing ppid=57 2023-04-12 15:56:07,407 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=58, ppid=57, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 490 msec 2023-04-12 15:56:07,408 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=57, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 906 msec 2023-04-12 15:56:07,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1227): Checking to see if procedure is done pid=57 2023-04-12 15:56:07,410 DEBUG [Listener at localhost/43583] zookeeper.ReadOnlyZKClient(139): Connect 0x4db22237 to 127.0.0.1:60154 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 15:56:07,416 DEBUG [Listener at localhost/43583] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@66b5b1f3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 15:56:07,416 DEBUG [Listener at localhost/43583] zookeeper.ReadOnlyZKClient(139): Connect 0x08b57aae to 127.0.0.1:60154 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 15:56:07,420 DEBUG [Listener at localhost/43583] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@544a4215, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 15:56:07,420 DEBUG [Listener at localhost/43583] zookeeper.ReadOnlyZKClient(139): Connect 0x06799455 to 127.0.0.1:60154 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 15:56:07,424 DEBUG [Listener at localhost/43583] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@25baee3e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 15:56:07,424 DEBUG [Listener at localhost/43583] zookeeper.ReadOnlyZKClient(139): Connect 0x66c25427 to 127.0.0.1:60154 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 15:56:07,427 DEBUG [Listener at localhost/43583] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2c53bf96, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 15:56:07,427 DEBUG [Listener at localhost/43583] zookeeper.ReadOnlyZKClient(139): Connect 0x4f4eab6d to 127.0.0.1:60154 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 15:56:07,432 DEBUG [Listener at localhost/43583] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3b59bdcf, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 15:56:07,432 DEBUG [Listener at localhost/43583] zookeeper.ReadOnlyZKClient(139): Connect 0x3d765193 to 127.0.0.1:60154 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 15:56:07,439 DEBUG [Listener at localhost/43583] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4bb72437, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 15:56:07,439 DEBUG [Listener at localhost/43583] zookeeper.ReadOnlyZKClient(139): Connect 0x32bf16cd to 127.0.0.1:60154 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 15:56:07,442 DEBUG [Listener at localhost/43583] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3dca23c0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 15:56:07,442 DEBUG [Listener at localhost/43583] zookeeper.ReadOnlyZKClient(139): Connect 0x28164190 to 127.0.0.1:60154 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 15:56:07,451 DEBUG [Listener at localhost/43583] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@62d52321, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 15:56:07,451 DEBUG [Listener at localhost/43583] zookeeper.ReadOnlyZKClient(139): Connect 0x308b3f39 to 127.0.0.1:60154 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 15:56:07,455 DEBUG [Listener at localhost/43583] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2a8bf279, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 15:56:07,456 DEBUG [Listener at localhost/43583] zookeeper.ReadOnlyZKClient(139): Connect 0x01453d93 to 127.0.0.1:60154 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 15:56:07,459 DEBUG [Listener at localhost/43583] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@68dea875, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 15:56:07,466 DEBUG [hconnection-0x4efbc8f8-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 15:56:07,467 DEBUG [hconnection-0xfcf7a76-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 15:56:07,468 DEBUG [hconnection-0x5f4bff32-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 15:56:07,469 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53320, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 15:56:07,473 DEBUG [hconnection-0x376f47b0-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 15:56:07,473 DEBUG [hconnection-0x24c02005-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 15:56:07,480 DEBUG [hconnection-0x6e9dbf27-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 15:56:07,480 DEBUG [hconnection-0x2e09cc4d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 15:56:07,481 DEBUG [hconnection-0x262c0759-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 15:56:07,481 DEBUG [hconnection-0x631c0c84-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 15:56:07,480 DEBUG [hconnection-0x2563cb95-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 15:56:07,482 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53306, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 15:56:07,482 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53292, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 15:56:07,483 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53332, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 15:56:07,483 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53324, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 15:56:07,483 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53344, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 15:56:07,484 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53360, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 15:56:07,485 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53378, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 15:56:07,487 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53364, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 15:56:07,490 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53392, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 15:56:07,490 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:56:07,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:56:07,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on 4870003209f366bc50ef2084e6e2cf03 2023-04-12 15:56:07,496 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4870003209f366bc50ef2084e6e2cf03 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-12 15:56:07,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:56:07,497 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=A 2023-04-12 15:56:07,497 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:07,497 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=B 2023-04-12 15:56:07,497 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:07,497 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=C 2023-04-12 15:56:07,497 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:07,497 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:56:07,497 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:07,497 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:56:07,497 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:56:07,497 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:07,497 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:07,499 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:07,499 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:07,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:07,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:07,499 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:07,499 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:56:07,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:07,499 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:07,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:56:07,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:07,500 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:56:07,500 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:56:07,500 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:56:07,500 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:56:07,500 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:07,500 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:56:07,504 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:56:07,504 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:56:07,504 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:07,504 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. started... 2023-04-12 15:56:07,504 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. as already flushing 2023-04-12 15:56:07,504 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:07,505 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:56:07,505 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:56:07,505 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:07,505 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:56:07,505 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:56:07,506 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:07,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:07,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:07,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:07,507 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:07,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:07,507 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:07,507 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:56:07,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:07,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:07,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:07,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:07,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:56:07,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5525affe[Count = 0] remaining members to acquire global barrier 2023-04-12 15:56:07,509 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:56:07,509 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:07,511 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:07,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:07,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:07,511 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:56:07,511 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:56:07,511 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:56:07,512 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:07,512 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:56:07,512 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:07,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53392 deadline: 1681315027510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:07,512 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:07,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53360 deadline: 1681315027510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:07,512 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:07,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53306 deadline: 1681315027512, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:07,513 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:56:07,513 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:07,513 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:56:07,513 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:07,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:07,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:07,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:07,514 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:07,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53324 deadline: 1681315027512, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:07,514 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:07,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:07,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53292 deadline: 1681315027512, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:07,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:07,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:07,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:07,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:07,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:07,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:07,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:56:07,517 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:56:07,517 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:56:07,517 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:56:07,517 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:56:07,517 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:56:07,517 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:56:07,519 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:07,519 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:07,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:07,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:07,519 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:07,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:07,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:07,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:07,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:07,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:07,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:07,520 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:07,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:07,521 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:07,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:07,522 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041280c9914cace340bf9df7fc71094c65ca to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041280c9914cace340bf9df7fc71094c65ca 2023-04-12 15:56:07,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:07,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:07,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:07,524 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:07,524 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:07,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:07,524 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:07,524 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:07,524 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:56:07,525 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:56:07,524 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:07,525 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:07,525 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:07,525 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:07,525 DEBUG [Thread-4110] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:56:07,525 DEBUG [Thread-4110] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:56:07,525 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:56:07,525 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:07,525 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:07,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:07,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:07,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:07,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:07,614 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:07,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53392 deadline: 1681315027613, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:07,615 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:07,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53360 deadline: 1681315027613, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:07,615 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:07,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53306 deadline: 1681315027613, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:07,617 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:07,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53324 deadline: 1681315027615, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:07,618 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:07,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53292 deadline: 1681315027615, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:07,625 DEBUG [Thread-4110] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:56:07,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:56:07,627 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:56:07,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:56:07,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:56:07,628 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:56:07,628 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:07,628 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:56:07,628 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:56:07,629 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:07,629 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:07,633 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:07,633 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:07,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:07,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:07,633 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:07,633 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:56:07,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:07,634 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:07,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:56:07,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:07,634 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:56:07,640 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:56:07,640 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:56:07,640 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:56:07,640 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:07,641 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:56:07,641 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:56:07,641 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:56:07,641 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:07,641 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. started... 2023-04-12 15:56:07,641 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. as already flushing 2023-04-12 15:56:07,641 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:07,641 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:56:07,641 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:56:07,641 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:07,641 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:56:07,641 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:56:07,645 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:07,645 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:07,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:07,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:07,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:07,646 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:07,646 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:56:07,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:07,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:07,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:07,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:07,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:07,647 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:56:07,647 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72c2d94d[Count = 0] remaining members to acquire global barrier 2023-04-12 15:56:07,647 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:56:07,647 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:07,649 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:07,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:07,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:07,649 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:56:07,649 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:56:07,649 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:07,649 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:56:07,649 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:56:07,651 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:07,651 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:56:07,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:07,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:07,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:07,651 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:07,652 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:56:07,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:07,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:07,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:07,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:07,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:07,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:07,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:07,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:56:07,654 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:56:07,654 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:56:07,654 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:56:07,654 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:56:07,654 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:56:07,654 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:56:07,656 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:07,656 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:07,656 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:07,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:07,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:07,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:07,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:07,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:07,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:07,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:07,656 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:07,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:07,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:07,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:07,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:07,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:07,658 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:07,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:07,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:07,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:07,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:07,660 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:07,660 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:07,660 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:07,660 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:07,660 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:07,660 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:56:07,661 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:56:07,660 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:07,661 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:07,661 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:07,661 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:07,660 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:07,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:07,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:07,661 DEBUG [Thread-4110] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:56:07,661 DEBUG [Thread-4110] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:56:07,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:07,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:07,761 DEBUG [Thread-4110] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:56:07,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:56:07,763 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:56:07,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:56:07,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:56:07,765 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:56:07,765 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:07,765 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:56:07,765 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:56:07,765 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:07,765 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:07,767 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:07,767 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:07,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:07,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:07,767 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:07,767 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:56:07,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:07,767 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:07,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:56:07,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:07,768 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:56:07,768 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:56:07,768 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:56:07,768 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:56:07,768 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:07,768 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:56:07,769 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:56:07,769 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:56:07,769 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:07,769 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. started... 2023-04-12 15:56:07,769 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. as already flushing 2023-04-12 15:56:07,769 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:07,770 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:56:07,770 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:56:07,770 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:07,770 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:56:07,770 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:56:07,772 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:07,772 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:07,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:07,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:07,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:07,772 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:07,772 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:56:07,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:07,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:07,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:07,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:07,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:07,774 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:56:07,774 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@786a7082[Count = 0] remaining members to acquire global barrier 2023-04-12 15:56:07,774 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:56:07,774 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:07,775 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:07,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:07,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:07,775 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:56:07,775 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:56:07,775 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:56:07,775 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:07,775 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:56:07,777 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:56:07,777 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:07,777 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:56:07,777 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:07,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:07,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:07,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:07,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:07,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:07,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:07,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:07,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:07,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:07,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:07,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:56:07,781 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:56:07,781 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:56:07,781 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:56:07,781 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:56:07,781 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:56:07,781 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:56:07,783 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:07,783 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:07,783 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:07,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:07,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:07,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:07,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:07,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:07,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:07,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:07,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:07,784 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:07,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:07,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:07,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:07,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:07,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:07,786 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:07,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:07,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:07,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:07,791 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:07,791 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:07,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:07,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:07,791 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:07,791 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:56:07,791 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:56:07,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:07,791 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:07,791 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:07,791 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:07,791 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:07,791 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:07,791 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:07,791 DEBUG [Thread-4110] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:56:07,791 DEBUG [Thread-4110] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:56:07,791 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:56:07,791 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:07,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:07,816 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:07,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53392 deadline: 1681315027816, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:07,816 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:07,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53360 deadline: 1681315027816, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:07,817 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:07,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53306 deadline: 1681315027816, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:07,819 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:07,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53324 deadline: 1681315027818, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:07,820 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:07,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53292 deadline: 1681315027819, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:07,891 DEBUG [Thread-4110] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:56:07,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:56:07,894 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:56:07,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:56:07,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:56:07,895 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:56:07,895 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:07,895 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:56:07,895 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:56:07,896 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:07,896 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:07,897 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:07,897 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:07,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:07,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:07,897 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:07,897 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:56:07,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:07,898 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:07,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:56:07,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:07,898 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:56:07,898 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:56:07,898 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:56:07,899 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:56:07,899 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:07,899 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:56:07,899 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:56:07,899 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:56:07,899 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:07,899 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. started... 2023-04-12 15:56:07,899 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. as already flushing 2023-04-12 15:56:07,899 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:07,899 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:56:07,899 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:56:07,899 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:07,899 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:56:07,899 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:56:07,901 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:07,901 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:07,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:07,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:07,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:07,901 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:07,901 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:56:07,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:07,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:07,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:07,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:07,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:07,903 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:56:07,903 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@547beee1[Count = 0] remaining members to acquire global barrier 2023-04-12 15:56:07,903 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:56:07,903 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:07,904 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:07,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:07,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:07,904 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:56:07,904 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:56:07,904 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:07,904 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:56:07,904 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:56:07,906 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:07,906 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:56:07,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:07,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:07,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:07,906 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:07,907 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:56:07,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:07,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:07,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:07,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:07,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:07,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:07,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:07,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:56:07,909 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:56:07,909 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:56:07,909 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:56:07,909 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:56:07,909 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:56:07,909 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:56:07,911 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:07,911 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:07,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:07,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:07,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:07,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:07,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:07,911 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:07,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:07,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:07,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:07,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:07,912 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:07,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:07,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:07,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:07,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:07,913 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:07,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:07,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:07,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:07,916 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:07,916 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:07,916 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:07,916 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:07,916 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:56:07,916 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:56:07,916 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:07,916 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:07,916 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:07,916 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:07,916 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:07,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:07,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:07,916 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:07,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:07,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:07,917 DEBUG [Thread-4110] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:56:07,917 DEBUG [Thread-4110] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:56:07,937 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=16, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/A/c3592a2ea53c4b34b78fbc83be19ebe2 2023-04-12 15:56:07,948 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/B/9c4128112654460982a183f69f08ee69 2023-04-12 15:56:07,957 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/C/53eb3046b978463496b92e9046a48f23 2023-04-12 15:56:07,961 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/A/c3592a2ea53c4b34b78fbc83be19ebe2 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/c3592a2ea53c4b34b78fbc83be19ebe2 2023-04-12 15:56:07,964 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/c3592a2ea53c4b34b78fbc83be19ebe2, entries=150, sequenceid=16, filesize=25.1 K 2023-04-12 15:56:07,965 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/B/9c4128112654460982a183f69f08ee69 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/9c4128112654460982a183f69f08ee69 2023-04-12 15:56:07,968 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/9c4128112654460982a183f69f08ee69, entries=150, sequenceid=16, filesize=11.6 K 2023-04-12 15:56:07,969 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/C/53eb3046b978463496b92e9046a48f23 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/53eb3046b978463496b92e9046a48f23 2023-04-12 15:56:07,972 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/53eb3046b978463496b92e9046a48f23, entries=150, sequenceid=16, filesize=11.6 K 2023-04-12 15:56:07,972 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 4870003209f366bc50ef2084e6e2cf03 in 476ms, sequenceid=16, compaction requested=false 2023-04-12 15:56:07,972 DEBUG [MemStoreFlusher.0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-12 15:56:07,973 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4870003209f366bc50ef2084e6e2cf03: 2023-04-12 15:56:08,017 DEBUG [Thread-4110] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:56:08,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:56:08,019 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:56:08,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:56:08,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:56:08,021 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:56:08,021 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:08,021 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:56:08,021 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:56:08,021 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:08,022 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:08,023 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:08,023 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:08,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:08,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:08,023 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:08,023 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:56:08,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:08,024 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:08,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:56:08,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:08,025 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:56:08,025 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:56:08,025 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:56:08,025 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:56:08,025 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:08,025 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:56:08,028 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:56:08,028 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:56:08,028 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:08,028 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. started... 2023-04-12 15:56:08,028 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4870003209f366bc50ef2084e6e2cf03 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-12 15:56:08,028 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=A 2023-04-12 15:56:08,028 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:08,029 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=B 2023-04-12 15:56:08,029 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:08,029 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=C 2023-04-12 15:56:08,029 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:08,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on 4870003209f366bc50ef2084e6e2cf03 2023-04-12 15:56:08,120 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. as already flushing 2023-04-12 15:56:08,127 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:08,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53392 deadline: 1681315028125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:08,127 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:08,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53324 deadline: 1681315028125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:08,128 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:08,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53292 deadline: 1681315028125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:08,128 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:08,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53306 deadline: 1681315028126, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:08,129 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:08,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53360 deadline: 1681315028127, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:08,229 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:08,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53392 deadline: 1681315028228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:08,230 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:08,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53324 deadline: 1681315028229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:08,230 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:08,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53306 deadline: 1681315028229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:08,232 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:08,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53292 deadline: 1681315028229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:08,232 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:08,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53360 deadline: 1681315028230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:08,432 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:08,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53392 deadline: 1681315028431, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:08,433 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:08,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53324 deadline: 1681315028432, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:08,433 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:08,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53306 deadline: 1681315028432, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:08,437 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:08,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53292 deadline: 1681315028437, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:08,437 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:08,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53360 deadline: 1681315028437, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:08,465 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304127e3ce1e0bded4442992c022d9ce06c4c to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304127e3ce1e0bded4442992c022d9ce06c4c 2023-04-12 15:56:08,471 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=36, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/A/14632b72474342a197ebfcab156ba9b6 2023-04-12 15:56:08,482 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=36 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/B/8a45af98195842cca6ac0d6a2904aea9 2023-04-12 15:56:08,514 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=36 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/C/e787be0dd18748b398a7f7864163748d 2023-04-12 15:56:08,519 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/A/14632b72474342a197ebfcab156ba9b6 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/14632b72474342a197ebfcab156ba9b6 2023-04-12 15:56:08,524 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/14632b72474342a197ebfcab156ba9b6, entries=150, sequenceid=36, filesize=25.1 K 2023-04-12 15:56:08,524 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/B/8a45af98195842cca6ac0d6a2904aea9 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/8a45af98195842cca6ac0d6a2904aea9 2023-04-12 15:56:08,528 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/8a45af98195842cca6ac0d6a2904aea9, entries=150, sequenceid=36, filesize=11.6 K 2023-04-12 15:56:08,529 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/C/e787be0dd18748b398a7f7864163748d as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/e787be0dd18748b398a7f7864163748d 2023-04-12 15:56:08,532 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/e787be0dd18748b398a7f7864163748d, entries=150, sequenceid=36, filesize=11.6 K 2023-04-12 15:56:08,533 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 4870003209f366bc50ef2084e6e2cf03 in 505ms, sequenceid=36, compaction requested=false 2023-04-12 15:56:08,537 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4870003209f366bc50ef2084e6e2cf03: 2023-04-12 15:56:08,537 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:08,537 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:56:08,537 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:56:08,537 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:08,537 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:56:08,537 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:56:08,539 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:08,539 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:08,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:08,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:08,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:08,540 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:08,540 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:56:08,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:08,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:08,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:08,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:08,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:08,542 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:56:08,542 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2cd9f1cf[Count = 0] remaining members to acquire global barrier 2023-04-12 15:56:08,542 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:56:08,542 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:08,546 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:08,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:08,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:08,546 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:56:08,546 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:56:08,546 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:56:08,546 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:08,547 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:56:08,547 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:56:08,547 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:08,547 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:08,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:08,547 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:56:08,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:08,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:08,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:08,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:08,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:08,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:08,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:08,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:08,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:08,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:56:08,550 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:56:08,550 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:56:08,550 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:56:08,550 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:56:08,550 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:56:08,550 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:56:08,553 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:08,553 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:08,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:08,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:08,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:08,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:08,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:08,554 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:08,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:08,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:08,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:08,554 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:08,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:08,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:08,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:08,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:08,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:08,557 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:08,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:08,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:08,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:08,560 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:08,560 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:08,560 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:08,560 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:08,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:08,560 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:56:08,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:08,560 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:08,560 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:08,560 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:08,560 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:08,560 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:08,560 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:56:08,560 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:56:08,560 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:08,561 DEBUG [Thread-4110] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:56:08,561 DEBUG [Thread-4110] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:56:08,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:08,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:08,661 DEBUG [Thread-4110] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:56:08,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:56:08,663 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:56:08,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:56:08,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:56:08,665 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:56:08,665 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:08,665 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:56:08,665 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:56:08,665 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:08,665 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:08,667 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:08,667 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:08,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:08,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:08,667 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:08,667 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:56:08,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:08,668 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:08,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:56:08,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:08,668 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:56:08,669 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:56:08,669 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:56:08,669 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:56:08,669 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:08,669 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:56:08,672 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:56:08,672 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:56:08,672 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:08,672 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. started... 2023-04-12 15:56:08,672 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4870003209f366bc50ef2084e6e2cf03 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-12 15:56:08,672 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=A 2023-04-12 15:56:08,672 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:08,672 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=B 2023-04-12 15:56:08,672 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:08,672 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=C 2023-04-12 15:56:08,672 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:08,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on 4870003209f366bc50ef2084e6e2cf03 2023-04-12 15:56:08,735 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. as already flushing 2023-04-12 15:56:08,755 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:08,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53306 deadline: 1681315028751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:08,756 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681314968746, max timestamp=1681314968755Num uniques 859; 2023-04-12 15:56:08,756 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-12 15:56:08,757 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681314968746, max timestamp=1681314968755Num uniques 859; 2023-04-12 15:56:08,757 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681314968746, max timestamp=1681314968755Num uniques 859; 2023-04-12 15:56:08,757 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-12 15:56:08,757 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-12 15:56:08,772 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:08,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53292 deadline: 1681315028769, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:08,773 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:08,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53392 deadline: 1681315028769, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:08,775 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:08,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53360 deadline: 1681315028773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:08,775 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:08,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53324 deadline: 1681315028773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:08,859 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:08,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53306 deadline: 1681315028856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:08,875 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:08,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53292 deadline: 1681315028874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:08,875 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:08,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53392 deadline: 1681315028874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:08,877 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:08,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53360 deadline: 1681315028876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:08,878 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:08,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53324 deadline: 1681315028876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:09,062 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:09,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53306 deadline: 1681315029061, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:09,079 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:09,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53292 deadline: 1681315029077, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:09,080 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:09,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53392 deadline: 1681315029078, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:09,080 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:09,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53360 deadline: 1681315029079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:09,082 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:09,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53324 deadline: 1681315029080, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:09,121 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041287deb95c1c504b42a8ff3215ae84aecf to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041287deb95c1c504b42a8ff3215ae84aecf 2023-04-12 15:56:09,126 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=48, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/A/648406cb77ba4910b5b3ca50c90ceffe 2023-04-12 15:56:09,365 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:09,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53306 deadline: 1681315029364, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:09,382 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:09,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53292 deadline: 1681315029381, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:09,384 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-12 15:56:09,386 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:09,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53392 deadline: 1681315029383, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:09,386 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:09,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53360 deadline: 1681315029383, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:09,386 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:09,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53324 deadline: 1681315029384, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:09,536 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/B/3cb02f3b11254f98bdf01e3f1ca8ec56 2023-04-12 15:56:09,547 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/C/a510f061d847430988389eae267a0dd8 2023-04-12 15:56:09,551 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/A/648406cb77ba4910b5b3ca50c90ceffe as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/648406cb77ba4910b5b3ca50c90ceffe 2023-04-12 15:56:09,561 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/648406cb77ba4910b5b3ca50c90ceffe, entries=100, sequenceid=48, filesize=18.4 K 2023-04-12 15:56:09,562 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/B/3cb02f3b11254f98bdf01e3f1ca8ec56 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/3cb02f3b11254f98bdf01e3f1ca8ec56 2023-04-12 15:56:09,569 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/3cb02f3b11254f98bdf01e3f1ca8ec56, entries=100, sequenceid=48, filesize=9.3 K 2023-04-12 15:56:09,570 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/C/a510f061d847430988389eae267a0dd8 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/a510f061d847430988389eae267a0dd8 2023-04-12 15:56:09,572 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/a510f061d847430988389eae267a0dd8, entries=100, sequenceid=48, filesize=9.3 K 2023-04-12 15:56:09,573 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=134.18 KB/137400 for 4870003209f366bc50ef2084e6e2cf03 in 901ms, sequenceid=48, compaction requested=true 2023-04-12 15:56:09,573 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4870003209f366bc50ef2084e6e2cf03: 2023-04-12 15:56:09,573 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:09,573 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:56:09,573 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:56:09,573 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:09,574 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:56:09,574 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:56:09,575 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:09,575 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:09,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:09,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:09,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:09,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:09,575 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:09,576 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:56:09,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:09,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:09,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:09,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:09,577 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:56:09,577 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48b54da8[Count = 0] remaining members to acquire global barrier 2023-04-12 15:56:09,577 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:56:09,577 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:09,579 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:09,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:09,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:09,579 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:56:09,579 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:56:09,579 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:56:09,580 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:09,580 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:56:09,581 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:56:09,581 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:09,581 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:09,581 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:56:09,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:09,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:09,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:09,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:09,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:09,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:09,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:09,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:09,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:09,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:09,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:56:09,584 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:56:09,584 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:56:09,584 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:56:09,584 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:56:09,584 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:56:09,584 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:56:09,586 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:09,586 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:09,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:09,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:09,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:09,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:09,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:09,586 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:09,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:09,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:09,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:09,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:09,587 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:09,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:09,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:09,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:09,588 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:09,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:09,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:09,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:09,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:09,590 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:09,591 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:09,591 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:09,591 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:09,591 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:09,591 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:09,591 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:56:09,591 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:56:09,591 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:09,591 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:09,591 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:09,591 DEBUG [Thread-4110] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:56:09,591 DEBUG [Thread-4110] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:56:09,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:09,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:09,591 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:09,591 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:56:09,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:09,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:09,691 DEBUG [Thread-4110] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:56:09,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:56:09,694 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:56:09,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:56:09,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:56:09,695 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:56:09,695 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:09,695 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:56:09,695 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:56:09,696 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:09,696 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:09,698 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:09,698 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:09,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:09,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:09,698 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:09,699 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:56:09,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:09,699 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:09,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:56:09,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:09,699 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:56:09,700 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:56:09,700 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:56:09,700 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:56:09,700 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:09,700 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:56:09,700 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:56:09,700 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:56:09,700 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:09,700 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. started... 2023-04-12 15:56:09,700 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4870003209f366bc50ef2084e6e2cf03 3/3 column families, dataSize=134.18 KB heapSize=370.92 KB 2023-04-12 15:56:09,701 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=A 2023-04-12 15:56:09,701 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:56:09,701 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=B 2023-04-12 15:56:09,701 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:56:09,701 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=C 2023-04-12 15:56:09,701 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:56:09,727 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041210e553c7e3e443db85a73234b10ce7da to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041210e553c7e3e443db85a73234b10ce7da 2023-04-12 15:56:09,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on 4870003209f366bc50ef2084e6e2cf03 2023-04-12 15:56:09,872 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. as already flushing 2023-04-12 15:56:09,890 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:09,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53392 deadline: 1681315029887, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:09,890 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:09,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53360 deadline: 1681315029887, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:09,893 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:09,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53306 deadline: 1681315029889, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:09,893 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:09,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53292 deadline: 1681315029890, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:09,893 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:09,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53324 deadline: 1681315029891, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:09,996 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:09,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53292 deadline: 1681315029994, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:09,996 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:09,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53306 deadline: 1681315029994, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:10,135 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=82, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/A/58e51f98e0724a8b8acf6d815d7e9d6b 2023-04-12 15:56:10,145 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/B/2d27599b9e95413aa6532d8109489593 2023-04-12 15:56:10,154 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/C/01b75d793f774d7bba66c689e6c002b6 2023-04-12 15:56:10,158 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/A/58e51f98e0724a8b8acf6d815d7e9d6b as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/58e51f98e0724a8b8acf6d815d7e9d6b 2023-04-12 15:56:10,162 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/58e51f98e0724a8b8acf6d815d7e9d6b, entries=150, sequenceid=82, filesize=25.1 K 2023-04-12 15:56:10,163 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/B/2d27599b9e95413aa6532d8109489593 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/2d27599b9e95413aa6532d8109489593 2023-04-12 15:56:10,166 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/2d27599b9e95413aa6532d8109489593, entries=150, sequenceid=82, filesize=11.6 K 2023-04-12 15:56:10,167 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/C/01b75d793f774d7bba66c689e6c002b6 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/01b75d793f774d7bba66c689e6c002b6 2023-04-12 15:56:10,169 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/01b75d793f774d7bba66c689e6c002b6, entries=150, sequenceid=82, filesize=11.6 K 2023-04-12 15:56:10,170 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~370.88 KB/379776, currentSize=53.67 KB/54960 for 4870003209f366bc50ef2084e6e2cf03 in 470ms, sequenceid=82, compaction requested=true 2023-04-12 15:56:10,170 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4870003209f366bc50ef2084e6e2cf03: 2023-04-12 15:56:10,170 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:10,170 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:56:10,170 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:56:10,170 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:10,171 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:56:10,171 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:56:10,172 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:10,172 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:10,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:10,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:10,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:10,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:10,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:10,173 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:10,173 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:56:10,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:10,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:10,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:10,175 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:56:10,175 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63f0ca5c[Count = 0] remaining members to acquire global barrier 2023-04-12 15:56:10,175 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:56:10,175 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:10,176 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:10,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:10,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:10,176 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:56:10,176 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:56:10,176 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:56:10,176 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:10,177 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:56:10,178 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:56:10,178 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:10,178 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:10,178 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:56:10,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:10,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:10,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:10,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:10,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:10,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:10,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:10,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:10,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:10,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:10,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:56:10,182 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:56:10,182 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:56:10,182 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:56:10,182 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:56:10,182 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:56:10,182 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:56:10,185 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:10,185 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:10,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:10,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:10,185 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:10,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:10,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:10,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:10,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:10,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:10,186 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:10,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:10,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:10,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:10,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:10,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:10,188 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:10,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:10,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:10,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:10,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:10,191 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:10,191 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:10,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:10,191 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:10,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:10,191 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:56:10,191 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:56:10,191 DEBUG [Thread-4110] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:56:10,191 DEBUG [Thread-4110] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:56:10,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:10,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:10,191 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:10,191 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:10,192 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:10,192 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:10,192 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:10,192 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:10,192 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:10,192 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:56:10,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on 4870003209f366bc50ef2084e6e2cf03 2023-04-12 15:56:10,200 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4870003209f366bc50ef2084e6e2cf03 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-12 15:56:10,201 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=A 2023-04-12 15:56:10,202 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:10,202 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=B 2023-04-12 15:56:10,202 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:10,202 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=C 2023-04-12 15:56:10,202 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:10,216 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230412568da4c0ad98472ea4c30475ae85658d to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412568da4c0ad98472ea4c30475ae85658d 2023-04-12 15:56:10,236 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:10,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53292 deadline: 1681315030234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:10,238 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:10,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53306 deadline: 1681315030236, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:10,291 DEBUG [Thread-4110] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:56:10,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:56:10,294 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:56:10,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:56:10,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:56:10,295 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:56:10,295 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:10,295 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:56:10,295 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:56:10,295 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:10,296 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:10,297 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:10,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:10,297 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:10,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:10,297 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:10,297 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:56:10,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:10,298 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:10,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:56:10,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:10,298 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:56:10,298 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:56:10,299 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:56:10,299 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:56:10,299 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:10,299 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:56:10,299 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:56:10,299 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:56:10,299 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:10,299 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. started... 2023-04-12 15:56:10,299 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. as already flushing 2023-04-12 15:56:10,299 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:10,299 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:56:10,299 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:56:10,299 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:10,299 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:56:10,299 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:56:10,301 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:10,301 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:10,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:10,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:10,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:10,301 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:10,301 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:56:10,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:10,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:10,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:10,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:10,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:10,302 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:56:10,303 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:56:10,303 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68f73c25[Count = 0] remaining members to acquire global barrier 2023-04-12 15:56:10,303 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:10,305 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:10,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:10,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:10,305 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:10,306 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:56:10,305 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:56:10,306 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:56:10,306 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:56:10,307 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:10,307 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:56:10,307 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:10,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:10,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:10,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:10,307 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:56:10,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:10,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:10,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:10,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:10,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:10,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:10,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:10,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:56:10,310 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:56:10,310 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:56:10,310 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:56:10,310 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:56:10,310 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:56:10,310 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:56:10,314 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:10,314 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:10,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:10,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:10,314 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:10,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:10,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:10,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:10,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:10,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:10,314 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:10,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:10,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:10,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:10,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:10,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:10,316 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:10,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:10,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:10,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:10,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:10,318 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:10,318 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:10,318 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:10,318 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:10,318 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:10,318 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:10,318 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:10,318 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:10,318 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:56:10,318 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:56:10,318 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:10,318 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:10,318 DEBUG [Thread-4110] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:56:10,319 DEBUG [Thread-4110] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:56:10,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:10,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:10,319 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:56:10,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:10,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:10,338 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:10,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53292 deadline: 1681315030337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:10,341 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:10,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53306 deadline: 1681315030339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:10,419 DEBUG [Thread-4110] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:56:10,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:56:10,421 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:56:10,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:56:10,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:56:10,422 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:56:10,422 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:10,422 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:56:10,422 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:56:10,423 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:10,423 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:10,425 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:10,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:10,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:10,425 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:10,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:10,426 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:10,426 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:56:10,426 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:10,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:56:10,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:10,426 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:56:10,427 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:56:10,427 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:56:10,427 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:56:10,427 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:10,427 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:56:10,427 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:56:10,427 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:56:10,428 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:10,428 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. started... 2023-04-12 15:56:10,428 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. as already flushing 2023-04-12 15:56:10,428 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:10,428 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:56:10,428 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:56:10,428 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:10,428 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:56:10,428 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:56:10,430 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:10,430 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:10,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:10,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:10,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:10,430 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:10,430 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:56:10,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:10,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:10,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:10,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:10,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:10,432 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:56:10,432 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b43ff7d[Count = 0] remaining members to acquire global barrier 2023-04-12 15:56:10,432 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:56:10,432 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:10,433 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:10,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:10,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:10,433 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:56:10,433 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:56:10,433 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:56:10,433 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:10,433 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:56:10,434 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:10,434 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:56:10,435 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:10,435 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:56:10,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:10,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:10,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:10,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:10,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:10,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:10,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:10,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:10,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:10,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:10,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:56:10,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:56:10,450 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:56:10,450 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:56:10,450 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:56:10,450 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:56:10,451 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:56:10,453 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:10,453 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:10,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:10,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:10,453 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:10,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:10,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:10,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:10,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:10,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:10,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:10,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:10,454 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:10,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:10,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:10,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:10,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:10,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:10,455 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:10,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:10,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:10,457 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:10,457 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:10,457 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:10,457 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:10,457 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:56:10,457 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:56:10,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:10,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:10,457 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:10,458 DEBUG [Thread-4110] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:56:10,457 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:10,458 DEBUG [Thread-4110] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:56:10,458 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:10,458 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:10,458 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:10,458 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:10,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:10,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:10,541 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:10,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53292 deadline: 1681315030539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:10,545 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:10,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53306 deadline: 1681315030543, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:10,558 DEBUG [Thread-4110] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:56:10,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:56:10,560 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:56:10,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:56:10,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:56:10,562 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:56:10,562 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:10,562 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:56:10,562 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:56:10,562 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:10,562 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:10,563 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:10,563 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:10,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:10,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:10,564 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:10,564 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:56:10,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:10,564 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:10,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:56:10,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:10,565 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:56:10,565 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:56:10,565 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:56:10,565 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:10,565 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:56:10,565 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:56:10,565 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:56:10,565 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:10,566 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. started... 2023-04-12 15:56:10,566 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. as already flushing 2023-04-12 15:56:10,566 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:10,566 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:56:10,566 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:56:10,566 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:10,566 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:56:10,566 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:56:10,567 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:10,567 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:10,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:10,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:10,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:10,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:10,568 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:10,568 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:56:10,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:10,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:10,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:10,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:10,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:56:10,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69792bb0[Count = 0] remaining members to acquire global barrier 2023-04-12 15:56:10,569 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:56:10,569 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:10,571 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:10,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:10,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:10,571 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:56:10,571 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:56:10,571 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:56:10,572 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:10,572 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:56:10,572 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:56:10,573 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:10,573 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:56:10,573 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:10,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:10,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:10,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:10,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:10,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:10,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:10,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:10,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:10,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:10,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:10,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:56:10,576 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:56:10,576 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:56:10,576 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:56:10,576 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:56:10,576 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:56:10,576 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:56:10,578 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:10,578 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:10,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:10,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:10,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:10,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:10,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:10,578 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:10,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:10,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:10,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:10,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:10,579 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:10,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:10,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:10,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:10,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:10,580 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:10,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:10,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:10,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:10,582 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:10,582 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:10,582 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:10,582 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:10,582 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:10,582 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:10,582 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:10,582 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:56:10,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:56:10,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:56:10,582 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:10,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:10,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:10,582 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:10,583 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:10,583 DEBUG [Thread-4110] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:56:10,583 DEBUG [Thread-4110] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:56:10,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:10,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:10,621 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=95, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/A/6f51d7e5811b4de6977036482e15daea 2023-04-12 15:56:10,633 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=95 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/B/e99449b3fbf6449e8c82abdd7a292380 2023-04-12 15:56:10,643 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=95 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/C/50e88dcadbd247818ceeaa717c4a5250 2023-04-12 15:56:10,646 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/A/6f51d7e5811b4de6977036482e15daea as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/6f51d7e5811b4de6977036482e15daea 2023-04-12 15:56:10,649 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/6f51d7e5811b4de6977036482e15daea, entries=200, sequenceid=95, filesize=31.9 K 2023-04-12 15:56:10,650 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/B/e99449b3fbf6449e8c82abdd7a292380 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/e99449b3fbf6449e8c82abdd7a292380 2023-04-12 15:56:10,653 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/e99449b3fbf6449e8c82abdd7a292380, entries=150, sequenceid=95, filesize=11.6 K 2023-04-12 15:56:10,654 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/C/50e88dcadbd247818ceeaa717c4a5250 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/50e88dcadbd247818ceeaa717c4a5250 2023-04-12 15:56:10,658 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/50e88dcadbd247818ceeaa717c4a5250, entries=150, sequenceid=95, filesize=11.6 K 2023-04-12 15:56:10,659 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 4870003209f366bc50ef2084e6e2cf03 in 459ms, sequenceid=95, compaction requested=true 2023-04-12 15:56:10,659 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4870003209f366bc50ef2084e6e2cf03: 2023-04-12 15:56:10,659 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:56:10,659 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-12 15:56:10,659 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-12 15:56:10,659 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:56:10,659 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:56:10,661 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 57086 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-12 15:56:10,661 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1912): 4870003209f366bc50ef2084e6e2cf03/B is initiating minor compaction (all files) 2023-04-12 15:56:10,661 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4870003209f366bc50ef2084e6e2cf03/B in TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:10,661 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 128722 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-12 15:56:10,661 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/9c4128112654460982a183f69f08ee69, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/8a45af98195842cca6ac0d6a2904aea9, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/3cb02f3b11254f98bdf01e3f1ca8ec56, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/2d27599b9e95413aa6532d8109489593, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/e99449b3fbf6449e8c82abdd7a292380] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp, totalSize=55.7 K 2023-04-12 15:56:10,661 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1912): 4870003209f366bc50ef2084e6e2cf03/A is initiating minor compaction (all files) 2023-04-12 15:56:10,662 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4870003209f366bc50ef2084e6e2cf03/A in TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:10,662 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/c3592a2ea53c4b34b78fbc83be19ebe2, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/14632b72474342a197ebfcab156ba9b6, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/648406cb77ba4910b5b3ca50c90ceffe, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/58e51f98e0724a8b8acf6d815d7e9d6b, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/6f51d7e5811b4de6977036482e15daea] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp, totalSize=125.7 K 2023-04-12 15:56:10,662 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 9c4128112654460982a183f69f08ee69, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1681314967493 2023-04-12 15:56:10,662 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting c3592a2ea53c4b34b78fbc83be19ebe2, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1681314967493 2023-04-12 15:56:10,662 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 8a45af98195842cca6ac0d6a2904aea9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1681314967509 2023-04-12 15:56:10,662 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 14632b72474342a197ebfcab156ba9b6, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1681314967509 2023-04-12 15:56:10,663 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 3cb02f3b11254f98bdf01e3f1ca8ec56, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1681314968125 2023-04-12 15:56:10,663 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 648406cb77ba4910b5b3ca50c90ceffe, keycount=100, bloomtype=ROW, size=18.4 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1681314968125 2023-04-12 15:56:10,663 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 2d27599b9e95413aa6532d8109489593, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1681314968767 2023-04-12 15:56:10,663 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 58e51f98e0724a8b8acf6d815d7e9d6b, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1681314968767 2023-04-12 15:56:10,663 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting e99449b3fbf6449e8c82abdd7a292380, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1681314969889 2023-04-12 15:56:10,664 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 6f51d7e5811b4de6977036482e15daea, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1681314969885 2023-04-12 15:56:10,672 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] throttle.PressureAwareThroughputController(145): 4870003209f366bc50ef2084e6e2cf03#B#compaction#568 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:56:10,678 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4870003209f366bc50ef2084e6e2cf03#A#compaction#569 average throughput is 3.29 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:56:10,683 DEBUG [Thread-4110] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:56:10,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:56:10,685 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:56:10,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:56:10,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:56:10,691 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:56:10,691 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:10,691 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:56:10,691 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:56:10,691 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:10,691 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:10,694 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:10,694 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:10,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:10,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:10,694 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:10,694 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:56:10,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:10,695 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:10,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:56:10,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:10,695 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:56:10,695 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:56:10,695 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:56:10,695 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:56:10,695 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:10,695 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:56:10,696 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:56:10,696 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:56:10,696 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:10,696 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. started... 2023-04-12 15:56:10,696 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4870003209f366bc50ef2084e6e2cf03 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-12 15:56:10,696 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=A 2023-04-12 15:56:10,696 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:10,696 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=B 2023-04-12 15:56:10,696 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:10,696 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=C 2023-04-12 15:56:10,696 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:10,730 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/A/e4d5a161e3b24a94a83578a20f1c850c as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/e4d5a161e3b24a94a83578a20f1c850c 2023-04-12 15:56:10,730 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041203499a3ac93946708751cdf0a419ccd7 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041203499a3ac93946708751cdf0a419ccd7 2023-04-12 15:56:10,735 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 4870003209f366bc50ef2084e6e2cf03/A of 4870003209f366bc50ef2084e6e2cf03 into e4d5a161e3b24a94a83578a20f1c850c(size=25.3 K), total size for store is 25.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:56:10,735 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4870003209f366bc50ef2084e6e2cf03: 2023-04-12 15:56:10,735 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03., storeName=4870003209f366bc50ef2084e6e2cf03/A, priority=11, startTime=1681314970659; duration=0sec 2023-04-12 15:56:10,735 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:56:10,735 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-12 15:56:10,736 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 57086 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-12 15:56:10,737 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1912): 4870003209f366bc50ef2084e6e2cf03/C is initiating minor compaction (all files) 2023-04-12 15:56:10,737 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4870003209f366bc50ef2084e6e2cf03/C in TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:10,737 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/53eb3046b978463496b92e9046a48f23, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/e787be0dd18748b398a7f7864163748d, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/a510f061d847430988389eae267a0dd8, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/01b75d793f774d7bba66c689e6c002b6, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/50e88dcadbd247818ceeaa717c4a5250] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp, totalSize=55.7 K 2023-04-12 15:56:10,737 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 53eb3046b978463496b92e9046a48f23, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1681314967493 2023-04-12 15:56:10,737 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting e787be0dd18748b398a7f7864163748d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1681314967509 2023-04-12 15:56:10,737 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting a510f061d847430988389eae267a0dd8, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1681314968125 2023-04-12 15:56:10,738 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 01b75d793f774d7bba66c689e6c002b6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1681314968767 2023-04-12 15:56:10,738 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 50e88dcadbd247818ceeaa717c4a5250, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1681314969889 2023-04-12 15:56:10,742 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=114, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/A/4c9ec657b3454f169fc44806f6254a70 2023-04-12 15:56:10,746 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4870003209f366bc50ef2084e6e2cf03#C#compaction#571 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:56:10,756 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/C/10e7fe83908e49d08e5fa67d9d15847a as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/10e7fe83908e49d08e5fa67d9d15847a 2023-04-12 15:56:10,760 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 4870003209f366bc50ef2084e6e2cf03/C of 4870003209f366bc50ef2084e6e2cf03 into 10e7fe83908e49d08e5fa67d9d15847a(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:56:10,760 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4870003209f366bc50ef2084e6e2cf03: 2023-04-12 15:56:10,760 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03., storeName=4870003209f366bc50ef2084e6e2cf03/C, priority=11, startTime=1681314970659; duration=0sec 2023-04-12 15:56:10,760 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:56:10,763 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=114 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/B/034bca029e534ba2b82e09fb77f6f21a 2023-04-12 15:56:10,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on 4870003209f366bc50ef2084e6e2cf03 2023-04-12 15:56:10,844 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. as already flushing 2023-04-12 15:56:10,890 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:10,890 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:10,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53306 deadline: 1681315030888, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:10,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53292 deadline: 1681315030889, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:10,897 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:10,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53360 deadline: 1681315030896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:10,898 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:10,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53324 deadline: 1681315030897, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:10,901 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:10,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53392 deadline: 1681315030900, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:10,993 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:10,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53306 deadline: 1681315030991, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:10,993 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:10,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53292 deadline: 1681315030991, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,094 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/B/8a1856f32e9e41e0903d31b684291e2e as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/8a1856f32e9e41e0903d31b684291e2e 2023-04-12 15:56:11,098 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 4870003209f366bc50ef2084e6e2cf03/B of 4870003209f366bc50ef2084e6e2cf03 into 8a1856f32e9e41e0903d31b684291e2e(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:56:11,098 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4870003209f366bc50ef2084e6e2cf03: 2023-04-12 15:56:11,098 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03., storeName=4870003209f366bc50ef2084e6e2cf03/B, priority=11, startTime=1681314970659; duration=0sec 2023-04-12 15:56:11,098 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:56:11,172 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=114 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/C/7eafe023391147ebbb60b87401bd5e2e 2023-04-12 15:56:11,176 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/A/4c9ec657b3454f169fc44806f6254a70 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/4c9ec657b3454f169fc44806f6254a70 2023-04-12 15:56:11,180 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/4c9ec657b3454f169fc44806f6254a70, entries=150, sequenceid=114, filesize=25.1 K 2023-04-12 15:56:11,181 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/B/034bca029e534ba2b82e09fb77f6f21a as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/034bca029e534ba2b82e09fb77f6f21a 2023-04-12 15:56:11,184 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/034bca029e534ba2b82e09fb77f6f21a, entries=150, sequenceid=114, filesize=11.6 K 2023-04-12 15:56:11,185 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/C/7eafe023391147ebbb60b87401bd5e2e as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/7eafe023391147ebbb60b87401bd5e2e 2023-04-12 15:56:11,188 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/7eafe023391147ebbb60b87401bd5e2e, entries=150, sequenceid=114, filesize=11.6 K 2023-04-12 15:56:11,189 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=73.80 KB/75570 for 4870003209f366bc50ef2084e6e2cf03 in 493ms, sequenceid=114, compaction requested=false 2023-04-12 15:56:11,189 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4870003209f366bc50ef2084e6e2cf03: 2023-04-12 15:56:11,189 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:11,189 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:56:11,189 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:56:11,189 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,189 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:56:11,189 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:56:11,191 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:11,191 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:11,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:11,191 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:11,191 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:56:11,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:11,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:11,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:11,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:11,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:56:11,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@cbebee9[Count = 0] remaining members to acquire global barrier 2023-04-12 15:56:11,193 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:56:11,193 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:11,196 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:11,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:11,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:11,196 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:56:11,196 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:56:11,196 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:56:11,197 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,197 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:56:11,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on 4870003209f366bc50ef2084e6e2cf03 2023-04-12 15:56:11,197 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4870003209f366bc50ef2084e6e2cf03 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-12 15:56:11,197 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=A 2023-04-12 15:56:11,197 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:56:11,197 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:11,197 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:11,198 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:56:11,198 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=B 2023-04-12 15:56:11,198 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:11,198 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,198 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=C 2023-04-12 15:56:11,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:11,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:11,198 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:11,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:11,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:11,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:11,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:11,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:11,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:56:11,203 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:56:11,203 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:56:11,203 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:56:11,203 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:56:11,203 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:56:11,203 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:56:11,206 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:11,206 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:11,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:11,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:11,206 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:11,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:11,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:11,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:11,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:11,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:11,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:11,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:11,207 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:11,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:11,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:11,209 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:11,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:11,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,214 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,214 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:11,214 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:11,214 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,214 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:11,214 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:11,214 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:11,215 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:11,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:11,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:11,214 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:11,215 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:11,215 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:56:11,215 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:56:11,215 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:56:11,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:11,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:11,215 DEBUG [Thread-4110] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:56:11,215 DEBUG [Thread-4110] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:56:11,219 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304127f2b3dcc8e39477387f1d2952809fb75 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304127f2b3dcc8e39477387f1d2952809fb75 2023-04-12 15:56:11,228 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:11,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53292 deadline: 1681315031224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,228 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:11,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53306 deadline: 1681315031225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,315 DEBUG [Thread-4110] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:56:11,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:56:11,317 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:56:11,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:56:11,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:56:11,321 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:56:11,321 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:11,321 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:56:11,321 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:56:11,321 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:11,321 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:11,323 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:11,323 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:11,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:11,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:11,323 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,323 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:56:11,324 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:11,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:56:11,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:11,324 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:56:11,325 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:56:11,325 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:56:11,325 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:56:11,325 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:11,325 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:56:11,325 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:56:11,325 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:56:11,325 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:11,325 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. started... 2023-04-12 15:56:11,325 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. as already flushing 2023-04-12 15:56:11,325 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:11,325 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:56:11,325 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:56:11,325 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,325 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:56:11,325 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:56:11,327 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:11,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:11,327 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:11,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:11,327 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:11,328 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:56:11,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:11,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:11,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:11,329 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:56:11,329 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5df9ad4f[Count = 0] remaining members to acquire global barrier 2023-04-12 15:56:11,329 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:56:11,329 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:11,330 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:11,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:11,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:11,330 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:56:11,330 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:56:11,330 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:56:11,331 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,331 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:56:11,331 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:11,331 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:11,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53306 deadline: 1681315031329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53292 deadline: 1681315031329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,336 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:56:11,336 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:11,336 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,336 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:56:11,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:11,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:11,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:11,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:11,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:11,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:11,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:11,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:56:11,338 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:56:11,338 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:56:11,338 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:56:11,338 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:56:11,338 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:56:11,338 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:56:11,340 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:11,340 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:11,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:11,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:11,340 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:11,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:11,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:11,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:11,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:11,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:11,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:11,341 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:11,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:11,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:11,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:11,342 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:11,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:11,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,344 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,344 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:11,344 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:11,344 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:11,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:11,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:11,344 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,344 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:11,345 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:11,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:11,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:11,345 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:56:11,345 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:56:11,345 DEBUG [Thread-4110] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:56:11,345 DEBUG [Thread-4110] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:56:11,345 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:11,345 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:11,345 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:11,345 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:56:11,445 DEBUG [Thread-4110] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:56:11,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:56:11,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:56:11,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:56:11,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:56:11,449 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:56:11,449 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:11,449 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:56:11,449 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:56:11,449 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:11,449 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:11,450 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:11,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:11,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:11,451 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:11,451 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,451 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:56:11,451 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:11,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:56:11,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:11,451 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:56:11,452 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:56:11,452 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:56:11,452 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:56:11,452 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:11,452 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:56:11,452 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:56:11,452 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:56:11,452 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:11,452 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. started... 2023-04-12 15:56:11,452 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. as already flushing 2023-04-12 15:56:11,452 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:11,452 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:56:11,452 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:56:11,452 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,452 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:56:11,453 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:56:11,455 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:11,455 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:11,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:11,455 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:11,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:11,455 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:56:11,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:11,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:11,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:11,456 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:56:11,456 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4955199a[Count = 0] remaining members to acquire global barrier 2023-04-12 15:56:11,456 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:56:11,456 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:11,458 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:11,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:11,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:11,458 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:56:11,458 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:56:11,458 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,458 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:56:11,458 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:56:11,459 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:56:11,459 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,459 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:11,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:11,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:11,460 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:56:11,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:11,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:11,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:11,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:11,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:11,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:56:11,462 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:56:11,462 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:56:11,462 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:56:11,462 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:56:11,462 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:56:11,462 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:56:11,464 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:11,464 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:11,464 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:11,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:11,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:11,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:11,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:11,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:11,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:11,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:11,464 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:11,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:11,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:11,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:11,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:11,466 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:11,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:11,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,469 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,469 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:11,469 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:11,469 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:11,469 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:11,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:11,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:11,469 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,469 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:11,469 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:56:11,469 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:56:11,469 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:11,469 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:11,469 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:11,469 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:56:11,469 DEBUG [Thread-4110] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:56:11,469 DEBUG [Thread-4110] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:56:11,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:11,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:11,533 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:11,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53306 deadline: 1681315031533, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,533 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:11,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53292 deadline: 1681315031533, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,570 DEBUG [Thread-4110] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:56:11,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:56:11,572 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:56:11,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:56:11,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:56:11,573 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:56:11,573 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:11,573 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:56:11,573 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:56:11,573 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:11,573 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:11,575 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:11,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:11,575 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:11,575 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,575 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:56:11,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:11,576 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:11,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:56:11,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:11,576 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:56:11,576 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:56:11,576 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:56:11,576 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:56:11,576 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:11,576 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:56:11,577 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:56:11,577 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:56:11,577 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:11,577 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. started... 2023-04-12 15:56:11,577 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. as already flushing 2023-04-12 15:56:11,577 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:11,577 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:56:11,577 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:56:11,577 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,577 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:56:11,577 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:56:11,579 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,579 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:11,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:11,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:11,579 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:11,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:11,579 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:56:11,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:11,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:11,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:11,581 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:56:11,581 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69518fab[Count = 0] remaining members to acquire global barrier 2023-04-12 15:56:11,581 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:56:11,581 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:11,582 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:11,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:11,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:11,582 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:56:11,582 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,583 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:56:11,582 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:56:11,583 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:56:11,585 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:56:11,585 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:11,585 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,585 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:56:11,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:11,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:11,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:11,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:11,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:11,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:11,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:11,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:56:11,587 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:56:11,587 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:56:11,588 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:56:11,588 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:56:11,588 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:56:11,588 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:56:11,589 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:11,589 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:11,589 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:11,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:11,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:11,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:11,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:11,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:11,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:11,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:11,590 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:11,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:11,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:11,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:11,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:11,591 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:11,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:11,596 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,596 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:11,596 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:11,596 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:11,596 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:11,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:11,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:11,596 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:11,596 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:56:11,596 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:56:11,596 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,596 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:11,597 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:11,597 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:11,597 DEBUG [Thread-4110] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:56:11,597 DEBUG [Thread-4110] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:56:11,597 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:56:11,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:11,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:11,634 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=133, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/A/071fc3308c3b41daa6b97b0e277421f0 2023-04-12 15:56:11,643 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=133 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/B/e00661d35ae74bf681fe4cbd8757bddf 2023-04-12 15:56:11,658 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=133 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/C/edd973233a354820888485ea48d455e1 2023-04-12 15:56:11,663 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/A/071fc3308c3b41daa6b97b0e277421f0 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/071fc3308c3b41daa6b97b0e277421f0 2023-04-12 15:56:11,669 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/071fc3308c3b41daa6b97b0e277421f0, entries=200, sequenceid=133, filesize=32.0 K 2023-04-12 15:56:11,669 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/B/e00661d35ae74bf681fe4cbd8757bddf as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/e00661d35ae74bf681fe4cbd8757bddf 2023-04-12 15:56:11,673 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/e00661d35ae74bf681fe4cbd8757bddf, entries=150, sequenceid=133, filesize=11.7 K 2023-04-12 15:56:11,674 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/C/edd973233a354820888485ea48d455e1 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/edd973233a354820888485ea48d455e1 2023-04-12 15:56:11,680 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/edd973233a354820888485ea48d455e1, entries=150, sequenceid=133, filesize=11.7 K 2023-04-12 15:56:11,681 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 4870003209f366bc50ef2084e6e2cf03 in 484ms, sequenceid=133, compaction requested=true 2023-04-12 15:56:11,681 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4870003209f366bc50ef2084e6e2cf03: 2023-04-12 15:56:11,681 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:56:11,681 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 15:56:11,681 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 15:56:11,681 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:56:11,681 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:56:11,682 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 84450 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 15:56:11,682 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35930 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 15:56:11,682 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1912): 4870003209f366bc50ef2084e6e2cf03/A is initiating minor compaction (all files) 2023-04-12 15:56:11,682 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1912): 4870003209f366bc50ef2084e6e2cf03/B is initiating minor compaction (all files) 2023-04-12 15:56:11,683 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4870003209f366bc50ef2084e6e2cf03/A in TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:11,683 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4870003209f366bc50ef2084e6e2cf03/B in TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:11,683 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/e4d5a161e3b24a94a83578a20f1c850c, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/4c9ec657b3454f169fc44806f6254a70, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/071fc3308c3b41daa6b97b0e277421f0] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp, totalSize=82.5 K 2023-04-12 15:56:11,683 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/8a1856f32e9e41e0903d31b684291e2e, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/034bca029e534ba2b82e09fb77f6f21a, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/e00661d35ae74bf681fe4cbd8757bddf] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp, totalSize=35.1 K 2023-04-12 15:56:11,683 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting e4d5a161e3b24a94a83578a20f1c850c, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1681314969889 2023-04-12 15:56:11,683 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 8a1856f32e9e41e0903d31b684291e2e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1681314969889 2023-04-12 15:56:11,683 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 4c9ec657b3454f169fc44806f6254a70, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=114, earliestPutTs=1681314970228 2023-04-12 15:56:11,683 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 034bca029e534ba2b82e09fb77f6f21a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=114, earliestPutTs=1681314970228 2023-04-12 15:56:11,684 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting e00661d35ae74bf681fe4cbd8757bddf, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=133, earliestPutTs=1681314970857 2023-04-12 15:56:11,684 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 071fc3308c3b41daa6b97b0e277421f0, keycount=200, bloomtype=ROW, size=32.0 K, encoding=NONE, compression=NONE, seqNum=133, earliestPutTs=1681314970857 2023-04-12 15:56:11,690 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] throttle.PressureAwareThroughputController(145): 4870003209f366bc50ef2084e6e2cf03#B#compaction#577 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:56:11,697 DEBUG [Thread-4110] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:56:11,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:56:11,697 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4870003209f366bc50ef2084e6e2cf03#A#compaction#578 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:56:11,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:56:11,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:56:11,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:56:11,704 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:56:11,704 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:11,704 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:56:11,704 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:56:11,704 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:11,705 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:11,707 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,707 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:11,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:11,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:11,707 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,708 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:56:11,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:11,708 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/B/0390a91420364d879a552226718d1d03 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/0390a91420364d879a552226718d1d03 2023-04-12 15:56:11,708 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:11,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:56:11,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:11,709 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:56:11,713 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4870003209f366bc50ef2084e6e2cf03/B of 4870003209f366bc50ef2084e6e2cf03 into 0390a91420364d879a552226718d1d03(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:56:11,713 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4870003209f366bc50ef2084e6e2cf03: 2023-04-12 15:56:11,713 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03., storeName=4870003209f366bc50ef2084e6e2cf03/B, priority=13, startTime=1681314971681; duration=0sec 2023-04-12 15:56:11,713 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:56:11,713 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 15:56:11,714 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35930 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 15:56:11,714 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1912): 4870003209f366bc50ef2084e6e2cf03/C is initiating minor compaction (all files) 2023-04-12 15:56:11,714 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4870003209f366bc50ef2084e6e2cf03/C in TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:11,714 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/10e7fe83908e49d08e5fa67d9d15847a, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/7eafe023391147ebbb60b87401bd5e2e, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/edd973233a354820888485ea48d455e1] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp, totalSize=35.1 K 2023-04-12 15:56:11,714 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 10e7fe83908e49d08e5fa67d9d15847a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1681314969889 2023-04-12 15:56:11,714 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 7eafe023391147ebbb60b87401bd5e2e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=114, earliestPutTs=1681314970228 2023-04-12 15:56:11,715 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting edd973233a354820888485ea48d455e1, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=133, earliestPutTs=1681314970857 2023-04-12 15:56:11,716 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:56:11,716 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:56:11,716 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:56:11,716 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:11,716 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:56:11,716 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:56:11,716 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:56:11,716 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:11,719 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. started... 2023-04-12 15:56:11,721 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4870003209f366bc50ef2084e6e2cf03 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-12 15:56:11,721 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=A 2023-04-12 15:56:11,721 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:11,722 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=B 2023-04-12 15:56:11,722 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:11,722 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=C 2023-04-12 15:56:11,722 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:11,725 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] throttle.PressureAwareThroughputController(145): 4870003209f366bc50ef2084e6e2cf03#C#compaction#579 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:56:11,743 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/C/c0567359de4546919d4c5915d62b0367 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/c0567359de4546919d4c5915d62b0367 2023-04-12 15:56:11,743 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230412fd2a3aa6dc5644d08758940050028903 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412fd2a3aa6dc5644d08758940050028903 2023-04-12 15:56:11,751 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4870003209f366bc50ef2084e6e2cf03/C of 4870003209f366bc50ef2084e6e2cf03 into c0567359de4546919d4c5915d62b0367(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:56:11,751 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4870003209f366bc50ef2084e6e2cf03: 2023-04-12 15:56:11,751 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03., storeName=4870003209f366bc50ef2084e6e2cf03/C, priority=13, startTime=1681314971681; duration=0sec 2023-04-12 15:56:11,751 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:56:11,756 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/A/6d5d115f609544f687aed070f1a61f0e as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/6d5d115f609544f687aed070f1a61f0e 2023-04-12 15:56:11,761 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4870003209f366bc50ef2084e6e2cf03/A of 4870003209f366bc50ef2084e6e2cf03 into 6d5d115f609544f687aed070f1a61f0e(size=25.5 K), total size for store is 25.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:56:11,762 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4870003209f366bc50ef2084e6e2cf03: 2023-04-12 15:56:11,762 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03., storeName=4870003209f366bc50ef2084e6e2cf03/A, priority=13, startTime=1681314971681; duration=0sec 2023-04-12 15:56:11,762 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:56:11,763 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=150, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/A/24cda499b21d47b59350bcd5728a426c 2023-04-12 15:56:11,787 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=150 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/B/10f0ccd9f5504b98b8977fec54d82d72 2023-04-12 15:56:11,799 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=150 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/C/50d1c61eb6c64c4b9801418dd9d5dc5a 2023-04-12 15:56:11,803 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/A/24cda499b21d47b59350bcd5728a426c as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/24cda499b21d47b59350bcd5728a426c 2023-04-12 15:56:11,806 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/24cda499b21d47b59350bcd5728a426c, entries=150, sequenceid=150, filesize=25.3 K 2023-04-12 15:56:11,807 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/B/10f0ccd9f5504b98b8977fec54d82d72 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/10f0ccd9f5504b98b8977fec54d82d72 2023-04-12 15:56:11,810 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/10f0ccd9f5504b98b8977fec54d82d72, entries=150, sequenceid=150, filesize=11.8 K 2023-04-12 15:56:11,811 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/C/50d1c61eb6c64c4b9801418dd9d5dc5a as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/50d1c61eb6c64c4b9801418dd9d5dc5a 2023-04-12 15:56:11,830 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/50d1c61eb6c64c4b9801418dd9d5dc5a, entries=150, sequenceid=150, filesize=11.8 K 2023-04-12 15:56:11,831 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=0 B/0 for 4870003209f366bc50ef2084e6e2cf03 in 109ms, sequenceid=150, compaction requested=false 2023-04-12 15:56:11,831 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4870003209f366bc50ef2084e6e2cf03: 2023-04-12 15:56:11,831 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:11,831 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:56:11,831 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:56:11,831 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,831 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:56:11,831 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:56:11,833 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:11,833 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:11,833 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:56:11,834 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:11,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:11,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:11,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:11,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:11,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:11,837 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:56:11,837 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10612d9e[Count = 0] remaining members to acquire global barrier 2023-04-12 15:56:11,837 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:56:11,837 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:11,838 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:11,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:11,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:11,838 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:56:11,838 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:56:11,838 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:56:11,840 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:56:11,840 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:11,840 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,840 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:56:11,840 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:56:11,840 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:56:11,840 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:56:11,840 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:56:11,840 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:56:11,840 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:56:11,841 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:56:11,843 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:11,843 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:11,843 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:11,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:11,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:11,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:11,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:11,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:11,843 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:11,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:11,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:11,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:11,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:11,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:11,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:11,845 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:11,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:11,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,847 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:11,847 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:11,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:11,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:11,847 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,847 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:56:11,847 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:56:11,847 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:11,848 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:56:11,848 DEBUG [Thread-4110] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:56:11,848 DEBUG [Thread-4110] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:56:11,847 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:11,848 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:11,848 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,848 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:11,848 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:11,848 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:11,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:11,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:11,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on 4870003209f366bc50ef2084e6e2cf03 2023-04-12 15:56:11,921 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4870003209f366bc50ef2084e6e2cf03 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-12 15:56:11,922 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=A 2023-04-12 15:56:11,922 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:11,922 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=B 2023-04-12 15:56:11,922 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:11,922 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=C 2023-04-12 15:56:11,922 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:11,948 DEBUG [Thread-4110] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:56:11,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:56:11,951 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:56:11,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:56:11,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:56:11,952 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:56:11,953 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:11,953 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:56:11,953 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:56:11,954 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:11,954 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:11,955 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:11,955 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:11,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:11,956 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,956 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:56:11,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:11,956 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:11,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:56:11,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:11,957 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:56:11,958 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:56:11,958 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:56:11,958 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:56:11,958 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:11,958 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:56:11,958 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:56:11,958 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:56:11,958 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:11,958 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. started... 2023-04-12 15:56:11,958 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. as already flushing 2023-04-12 15:56:11,958 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:11,958 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:56:11,958 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:56:11,959 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,959 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:56:11,959 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:56:11,960 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,960 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:11,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:11,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:11,961 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:11,961 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:56:11,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:11,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:11,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:11,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:11,962 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:56:11,963 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:56:11,963 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:11,963 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e27908[Count = 0] remaining members to acquire global barrier 2023-04-12 15:56:11,963 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681314971950, max timestamp=1681314971962Num uniques 859; 2023-04-12 15:56:11,963 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-12 15:56:11,964 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681314971950, max timestamp=1681314971962Num uniques 859; 2023-04-12 15:56:11,964 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-12 15:56:11,964 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681314971950, max timestamp=1681314971962Num uniques 859; 2023-04-12 15:56:11,964 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-12 15:56:11,964 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:11,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:11,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:11,964 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,965 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:56:11,965 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:56:11,965 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:56:11,965 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:56:11,966 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,966 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:56:11,966 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:11,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:11,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:11,966 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:56:11,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:11,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:11,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:11,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:11,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:11,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:56:11,969 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:56:11,969 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:56:11,970 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:56:11,970 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:56:11,970 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:56:11,970 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:56:11,971 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:11,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:11,972 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:11,972 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:11,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:11,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:11,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:11,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:11,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:11,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:11,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:11,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:11,973 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:11,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:11,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:11,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,974 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:11,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:11,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,976 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,976 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:11,977 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:11,977 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:11,977 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:11,977 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:11,977 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:11,977 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:11,977 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:11,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:11,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:11,977 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:11,977 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:56:11,977 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:56:11,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:11,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:11,977 DEBUG [Thread-4110] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:56:11,977 DEBUG [Thread-4110] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:56:12,019 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:12,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53292 deadline: 1681315032017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,021 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:12,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53306 deadline: 1681315032019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,077 DEBUG [Thread-4110] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:56:12,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:56:12,079 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:56:12,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:56:12,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:56:12,080 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:56:12,081 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:12,081 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:56:12,081 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:56:12,081 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:12,081 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:12,083 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:12,083 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:12,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:12,084 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,084 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:56:12,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:12,084 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:12,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:56:12,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:12,085 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:56:12,085 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:56:12,085 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:56:12,085 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:56:12,085 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:12,085 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:56:12,085 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:56:12,085 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:56:12,085 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:12,085 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. started... 2023-04-12 15:56:12,085 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. as already flushing 2023-04-12 15:56:12,085 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:12,086 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:56:12,086 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:56:12,086 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,086 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:56:12,086 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:56:12,087 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:12,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:12,087 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:12,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:12,088 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:12,088 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:56:12,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:12,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:12,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:12,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:56:12,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@611a3947[Count = 0] remaining members to acquire global barrier 2023-04-12 15:56:12,089 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:56:12,090 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:12,091 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:12,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:12,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:12,091 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:56:12,091 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:56:12,091 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:56:12,091 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,091 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:56:12,093 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:56:12,093 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:12,093 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,093 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:56:12,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:12,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:12,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:12,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:12,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:12,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:12,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:12,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:56:12,096 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:56:12,096 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:56:12,096 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:56:12,096 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:56:12,096 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:56:12,096 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:56:12,099 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:12,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:12,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:12,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:12,099 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:12,099 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:12,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:12,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:12,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:12,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:12,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:12,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:12,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:12,100 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:12,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:12,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,102 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:12,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:12,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,104 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,104 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:12,104 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:12,104 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,104 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:12,104 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:12,104 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:12,104 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:12,104 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:12,104 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:56:12,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:12,104 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:56:12,105 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:56:12,104 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:12,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:12,105 DEBUG [Thread-4110] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:56:12,105 DEBUG [Thread-4110] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:56:12,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:12,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:12,121 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:12,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53292 deadline: 1681315032120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,122 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:12,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53306 deadline: 1681315032122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,205 DEBUG [Thread-4110] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:56:12,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:56:12,207 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:56:12,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:56:12,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:56:12,209 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:56:12,209 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:12,209 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:56:12,209 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:56:12,210 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:12,210 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:12,211 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,211 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:12,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:12,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:12,212 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,212 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:56:12,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:12,212 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:12,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:56:12,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:12,212 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:56:12,213 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:56:12,213 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:56:12,213 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:56:12,213 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:12,213 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:56:12,213 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:56:12,213 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:56:12,213 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:12,213 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. started... 2023-04-12 15:56:12,213 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. as already flushing 2023-04-12 15:56:12,213 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:12,213 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:56:12,214 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:56:12,214 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,214 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:56:12,214 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:56:12,219 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,219 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:12,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:12,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:12,220 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:12,220 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:56:12,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:12,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:12,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:12,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:12,221 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:56:12,221 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@453de18[Count = 0] remaining members to acquire global barrier 2023-04-12 15:56:12,221 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:56:12,221 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:12,222 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:12,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:12,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:12,222 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:56:12,223 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:56:12,223 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,223 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:56:12,223 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:56:12,224 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:56:12,224 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:12,224 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,224 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:56:12,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:12,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:12,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:12,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:12,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:12,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:12,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:12,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:56:12,227 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:56:12,227 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:56:12,227 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:56:12,227 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:56:12,227 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:56:12,227 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:56:12,228 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:12,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:12,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:12,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:12,228 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:12,229 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:12,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:12,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:12,229 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:12,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:12,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:12,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:12,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:12,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:12,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:12,230 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:12,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:12,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,233 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,233 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:12,233 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:12,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:12,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:12,234 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:12,233 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:12,233 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:12,234 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,234 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:56:12,234 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:56:12,234 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:56:12,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:12,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:12,234 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:12,234 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:12,234 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:12,234 DEBUG [Thread-4110] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:56:12,234 DEBUG [Thread-4110] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:56:12,323 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:12,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53292 deadline: 1681315032323, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,325 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:12,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53306 deadline: 1681315032323, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,334 DEBUG [Thread-4110] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:56:12,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:56:12,336 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:56:12,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:56:12,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:56:12,337 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:56:12,337 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:12,337 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:56:12,338 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:56:12,338 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:12,338 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:12,339 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:12,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:12,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:12,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:12,340 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,340 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:12,340 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,340 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:56:12,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:56:12,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:12,341 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:56:12,341 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:56:12,341 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:56:12,341 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:56:12,341 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:12,341 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:56:12,342 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:56:12,342 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:56:12,342 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:12,342 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. started... 2023-04-12 15:56:12,342 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. as already flushing 2023-04-12 15:56:12,342 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:12,342 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:56:12,342 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:56:12,342 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,342 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:56:12,342 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:56:12,342 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230412c1d6542811424cf0ba96cdbfacb3fd84 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412c1d6542811424cf0ba96cdbfacb3fd84 2023-04-12 15:56:12,345 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:12,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:12,345 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:12,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:12,345 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:12,345 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:56:12,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:12,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:12,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:12,347 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:56:12,347 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1dc71ed1[Count = 0] remaining members to acquire global barrier 2023-04-12 15:56:12,347 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:56:12,347 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:12,349 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:12,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:12,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:12,349 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:56:12,349 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:56:12,349 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:56:12,349 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,349 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:56:12,351 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,351 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:56:12,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:12,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:12,351 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:12,352 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:56:12,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:12,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:12,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:12,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:12,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:12,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:56:12,354 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:56:12,354 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:56:12,354 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:56:12,354 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:56:12,354 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:56:12,354 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:56:12,356 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:12,356 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:12,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:12,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:12,356 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:12,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:12,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:12,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:12,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:12,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:12,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:12,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:12,357 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:12,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:12,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:12,359 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:12,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:12,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,361 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:12,361 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:12,361 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:12,361 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:12,361 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:56:12,362 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:56:12,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:12,361 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:12,362 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:12,362 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,362 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:12,362 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:12,362 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:12,362 DEBUG [Thread-4110] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:56:12,362 DEBUG [Thread-4110] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:56:12,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:12,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:12,462 DEBUG [Thread-4110] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:56:12,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:56:12,465 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:56:12,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:56:12,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:56:12,466 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:56:12,466 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:12,466 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:56:12,466 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:56:12,466 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:12,466 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:12,468 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,468 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:12,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:12,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:12,468 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,468 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:56:12,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:12,469 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:12,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:56:12,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:12,469 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:56:12,469 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:56:12,469 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:56:12,469 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:56:12,469 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:12,469 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:56:12,470 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:56:12,470 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:56:12,470 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:12,470 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. started... 2023-04-12 15:56:12,470 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. as already flushing 2023-04-12 15:56:12,470 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:12,470 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:56:12,470 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:56:12,470 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,470 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:56:12,470 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:56:12,472 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,472 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:12,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:12,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:12,473 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:12,473 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:56:12,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:12,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:12,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:12,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:12,474 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:56:12,474 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:56:12,474 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e7e0e58[Count = 0] remaining members to acquire global barrier 2023-04-12 15:56:12,474 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:12,475 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:12,475 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,476 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:56:12,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:12,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:12,476 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:56:12,476 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:56:12,476 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:56:12,477 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,477 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:56:12,477 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:12,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:12,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:12,477 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:56:12,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:12,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:12,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:12,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:12,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:12,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:56:12,480 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:56:12,480 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:56:12,480 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:56:12,480 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:56:12,480 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:56:12,480 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:56:12,481 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:12,481 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:12,481 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:12,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:12,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:12,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:12,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:12,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:12,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:12,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:12,482 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:12,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:12,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:12,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:12,483 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:12,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:12,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:12,486 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,486 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:12,486 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:12,486 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:12,486 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:12,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:12,486 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:12,486 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:56:12,486 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:56:12,486 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:56:12,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:12,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:12,486 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:12,486 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:12,487 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:12,487 DEBUG [Thread-4110] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:56:12,487 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:12,487 DEBUG [Thread-4110] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:56:12,587 DEBUG [Thread-4110] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:56:12,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:56:12,589 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:56:12,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:56:12,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:56:12,591 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:56:12,591 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:12,591 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:56:12,591 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:56:12,592 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:12,592 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:12,593 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,593 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:12,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:12,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:12,593 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,593 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:56:12,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:12,594 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:12,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:56:12,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:12,594 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:56:12,594 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:56:12,594 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:56:12,594 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:56:12,594 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:12,594 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:56:12,595 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:56:12,595 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:56:12,595 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:12,595 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. started... 2023-04-12 15:56:12,595 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. as already flushing 2023-04-12 15:56:12,595 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:12,595 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:56:12,595 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:56:12,595 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,595 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:56:12,595 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:56:12,597 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:12,597 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:12,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:12,597 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:12,597 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:56:12,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:12,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:12,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:12,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:12,598 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:56:12,598 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e9eced1[Count = 0] remaining members to acquire global barrier 2023-04-12 15:56:12,598 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:56:12,599 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:12,602 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:12,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:12,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:12,602 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:56:12,602 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:56:12,602 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:56:12,603 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,603 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:56:12,604 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,604 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:56:12,604 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:12,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:12,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:12,604 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:56:12,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:12,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:12,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:12,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:12,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:12,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:56:12,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:56:12,606 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:56:12,606 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:56:12,606 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:56:12,606 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:56:12,606 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:56:12,608 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:12,608 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:12,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:12,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:12,608 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:12,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:12,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:12,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:12,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:12,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:12,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:12,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:12,609 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:12,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:12,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:12,610 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:12,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:12,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,612 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,612 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:12,612 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:12,612 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,612 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:12,612 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:12,612 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:12,612 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:56:12,612 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:12,612 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:12,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:12,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:12,612 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:56:12,612 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:56:12,612 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:12,613 DEBUG [Thread-4110] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:56:12,613 DEBUG [Thread-4110] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:56:12,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:12,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:12,638 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:12,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53292 deadline: 1681315032624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,641 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:12,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53306 deadline: 1681315032637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,713 DEBUG [Thread-4110] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:56:12,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:56:12,715 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:56:12,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:56:12,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:56:12,716 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:56:12,716 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:12,716 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:56:12,716 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:56:12,717 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:12,717 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:12,719 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:12,719 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:12,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:12,719 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,719 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:56:12,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:12,720 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:12,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:56:12,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:12,720 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:56:12,720 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:56:12,720 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:56:12,720 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:12,720 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:56:12,720 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:56:12,720 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:56:12,721 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:12,721 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. started... 2023-04-12 15:56:12,721 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. as already flushing 2023-04-12 15:56:12,721 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:12,721 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:56:12,721 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:56:12,721 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,721 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:56:12,721 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:56:12,723 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:12,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:12,723 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:12,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:12,723 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:12,724 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:56:12,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:12,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:12,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:12,725 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:56:12,725 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24276404[Count = 0] remaining members to acquire global barrier 2023-04-12 15:56:12,725 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:56:12,725 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:12,726 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:12,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:12,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:12,727 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:56:12,727 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:56:12,727 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:56:12,727 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,727 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:56:12,728 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,728 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:56:12,728 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:12,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:12,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:12,728 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:56:12,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:12,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:12,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:12,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:12,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:12,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:56:12,731 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:56:12,731 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:56:12,731 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:56:12,731 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:56:12,731 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:56:12,731 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:56:12,734 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:12,734 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:12,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:12,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:12,734 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:12,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:12,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:12,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:12,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:12,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:12,735 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:12,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:12,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:12,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:12,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:12,736 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:12,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:12,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,738 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:12,738 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:12,738 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:56:12,738 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:56:12,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:12,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:12,738 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,738 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:12,738 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:12,738 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:12,738 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,738 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:12,738 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:12,738 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:12,738 DEBUG [Thread-4110] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:56:12,739 DEBUG [Thread-4110] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:56:12,739 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:56:12,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:12,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:12,749 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=163, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/A/6c1ff151010b46c4a3dd84812f4069b4 2023-04-12 15:56:12,758 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=163 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/B/93f7f06f96324b8c9d4d5a49df0c99b9 2023-04-12 15:56:12,767 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=163 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/C/31e495d15d5b4f41b259a39cb5581962 2023-04-12 15:56:12,770 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/A/6c1ff151010b46c4a3dd84812f4069b4 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/6c1ff151010b46c4a3dd84812f4069b4 2023-04-12 15:56:12,774 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/6c1ff151010b46c4a3dd84812f4069b4, entries=300, sequenceid=163, filesize=45.8 K 2023-04-12 15:56:12,774 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/B/93f7f06f96324b8c9d4d5a49df0c99b9 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/93f7f06f96324b8c9d4d5a49df0c99b9 2023-04-12 15:56:12,777 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/93f7f06f96324b8c9d4d5a49df0c99b9, entries=150, sequenceid=163, filesize=11.8 K 2023-04-12 15:56:12,778 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/C/31e495d15d5b4f41b259a39cb5581962 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/31e495d15d5b4f41b259a39cb5581962 2023-04-12 15:56:12,781 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/31e495d15d5b4f41b259a39cb5581962, entries=150, sequenceid=163, filesize=11.8 K 2023-04-12 15:56:12,781 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 4870003209f366bc50ef2084e6e2cf03 in 860ms, sequenceid=163, compaction requested=true 2023-04-12 15:56:12,782 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4870003209f366bc50ef2084e6e2cf03: 2023-04-12 15:56:12,782 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:56:12,782 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 15:56:12,782 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 15:56:12,782 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:56:12,782 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:56:12,783 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 98886 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 15:56:12,783 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1912): 4870003209f366bc50ef2084e6e2cf03/A is initiating minor compaction (all files) 2023-04-12 15:56:12,783 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4870003209f366bc50ef2084e6e2cf03/A in TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:12,783 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/6d5d115f609544f687aed070f1a61f0e, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/24cda499b21d47b59350bcd5728a426c, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/6c1ff151010b46c4a3dd84812f4069b4] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp, totalSize=96.6 K 2023-04-12 15:56:12,784 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36332 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 15:56:12,784 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1912): 4870003209f366bc50ef2084e6e2cf03/B is initiating minor compaction (all files) 2023-04-12 15:56:12,784 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4870003209f366bc50ef2084e6e2cf03/B in TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:12,784 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/0390a91420364d879a552226718d1d03, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/10f0ccd9f5504b98b8977fec54d82d72, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/93f7f06f96324b8c9d4d5a49df0c99b9] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp, totalSize=35.5 K 2023-04-12 15:56:12,784 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 6d5d115f609544f687aed070f1a61f0e, keycount=150, bloomtype=ROW, size=25.5 K, encoding=NONE, compression=NONE, seqNum=133, earliestPutTs=1681314970857 2023-04-12 15:56:12,784 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 0390a91420364d879a552226718d1d03, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=133, earliestPutTs=1681314970857 2023-04-12 15:56:12,784 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 24cda499b21d47b59350bcd5728a426c, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=150, earliestPutTs=1681314971208 2023-04-12 15:56:12,785 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 10f0ccd9f5504b98b8977fec54d82d72, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=150, earliestPutTs=1681314971208 2023-04-12 15:56:12,785 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 6c1ff151010b46c4a3dd84812f4069b4, keycount=300, bloomtype=ROW, size=45.8 K, encoding=NONE, compression=NONE, seqNum=163, earliestPutTs=1681314971865 2023-04-12 15:56:12,785 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 93f7f06f96324b8c9d4d5a49df0c99b9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=163, earliestPutTs=1681314971865 2023-04-12 15:56:12,793 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] throttle.PressureAwareThroughputController(145): 4870003209f366bc50ef2084e6e2cf03#B#compaction#587 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:56:12,801 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4870003209f366bc50ef2084e6e2cf03#A#compaction#586 average throughput is 1.97 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:56:12,839 DEBUG [Thread-4110] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:56:12,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:56:12,841 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:56:12,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:56:12,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:56:12,842 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:56:12,842 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:12,842 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:56:12,842 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:56:12,843 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:12,843 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:12,844 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:12,844 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:12,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:12,844 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,844 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:56:12,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:12,845 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:12,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:56:12,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:12,845 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:56:12,846 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:56:12,846 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:56:12,846 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:56:12,846 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:12,846 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:56:12,846 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:56:12,846 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:56:12,847 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:12,847 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. started... 2023-04-12 15:56:12,847 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4870003209f366bc50ef2084e6e2cf03 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-12 15:56:12,847 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=A 2023-04-12 15:56:12,847 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:56:12,847 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=B 2023-04-12 15:56:12,847 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:56:12,847 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=C 2023-04-12 15:56:12,847 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:56:12,860 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230412d09eeced51bf4fa197d583085e635a1b to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412d09eeced51bf4fa197d583085e635a1b 2023-04-12 15:56:12,865 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=198, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/A/803c0ebb03de497597a4fb2603433b4c 2023-04-12 15:56:12,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on 4870003209f366bc50ef2084e6e2cf03 2023-04-12 15:56:12,911 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. as already flushing 2023-04-12 15:56:12,928 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:12,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53324 deadline: 1681315032925, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,931 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:12,931 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:12,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53360 deadline: 1681315032927, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:12,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53392 deadline: 1681315032927, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:13,032 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:13,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53324 deadline: 1681315033029, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:13,037 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:13,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53392 deadline: 1681315033033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:13,037 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:13,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53360 deadline: 1681315033033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:13,145 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:13,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53292 deadline: 1681315033141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:13,145 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:13,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53306 deadline: 1681315033143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:13,202 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/B/b01faf10215543e89ab551370b386ccc as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/b01faf10215543e89ab551370b386ccc 2023-04-12 15:56:13,207 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4870003209f366bc50ef2084e6e2cf03/B of 4870003209f366bc50ef2084e6e2cf03 into b01faf10215543e89ab551370b386ccc(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:56:13,207 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4870003209f366bc50ef2084e6e2cf03: 2023-04-12 15:56:13,207 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03., storeName=4870003209f366bc50ef2084e6e2cf03/B, priority=13, startTime=1681314972782; duration=0sec 2023-04-12 15:56:13,207 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:56:13,207 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 15:56:13,208 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36332 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 15:56:13,208 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1912): 4870003209f366bc50ef2084e6e2cf03/C is initiating minor compaction (all files) 2023-04-12 15:56:13,208 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4870003209f366bc50ef2084e6e2cf03/C in TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:13,208 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/c0567359de4546919d4c5915d62b0367, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/50d1c61eb6c64c4b9801418dd9d5dc5a, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/31e495d15d5b4f41b259a39cb5581962] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp, totalSize=35.5 K 2023-04-12 15:56:13,209 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting c0567359de4546919d4c5915d62b0367, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=133, earliestPutTs=1681314970857 2023-04-12 15:56:13,209 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 50d1c61eb6c64c4b9801418dd9d5dc5a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=150, earliestPutTs=1681314971208 2023-04-12 15:56:13,210 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 31e495d15d5b4f41b259a39cb5581962, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=163, earliestPutTs=1681314971865 2023-04-12 15:56:13,215 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] throttle.PressureAwareThroughputController(145): 4870003209f366bc50ef2084e6e2cf03#C#compaction#590 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:56:13,225 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/C/d76c5cda3a31405997aa8220567caf75 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/d76c5cda3a31405997aa8220567caf75 2023-04-12 15:56:13,229 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/A/6cf94b918db84e4da8056344fdb7def6 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/6cf94b918db84e4da8056344fdb7def6 2023-04-12 15:56:13,231 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4870003209f366bc50ef2084e6e2cf03/C of 4870003209f366bc50ef2084e6e2cf03 into d76c5cda3a31405997aa8220567caf75(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:56:13,231 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4870003209f366bc50ef2084e6e2cf03: 2023-04-12 15:56:13,231 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03., storeName=4870003209f366bc50ef2084e6e2cf03/C, priority=13, startTime=1681314972782; duration=0sec 2023-04-12 15:56:13,231 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:56:13,234 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4870003209f366bc50ef2084e6e2cf03/A of 4870003209f366bc50ef2084e6e2cf03 into 6cf94b918db84e4da8056344fdb7def6(size=25.7 K), total size for store is 25.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:56:13,235 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4870003209f366bc50ef2084e6e2cf03: 2023-04-12 15:56:13,235 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03., storeName=4870003209f366bc50ef2084e6e2cf03/A, priority=13, startTime=1681314972782; duration=0sec 2023-04-12 15:56:13,235 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:13,235 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:56:13,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53324 deadline: 1681315033233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:13,238 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:13,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53392 deadline: 1681315033238, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:13,241 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:13,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53360 deadline: 1681315033239, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:13,298 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=198 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/B/70215b26de4541da8cd467d0489e6a9b 2023-04-12 15:56:13,307 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=198 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/C/96a6307aae664f5a9d1863cdbddd6080 2023-04-12 15:56:13,311 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/A/803c0ebb03de497597a4fb2603433b4c as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/803c0ebb03de497597a4fb2603433b4c 2023-04-12 15:56:13,315 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/803c0ebb03de497597a4fb2603433b4c, entries=150, sequenceid=198, filesize=25.3 K 2023-04-12 15:56:13,315 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/B/70215b26de4541da8cd467d0489e6a9b as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/70215b26de4541da8cd467d0489e6a9b 2023-04-12 15:56:13,318 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/70215b26de4541da8cd467d0489e6a9b, entries=150, sequenceid=198, filesize=11.8 K 2023-04-12 15:56:13,319 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/C/96a6307aae664f5a9d1863cdbddd6080 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/96a6307aae664f5a9d1863cdbddd6080 2023-04-12 15:56:13,324 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/96a6307aae664f5a9d1863cdbddd6080, entries=150, sequenceid=198, filesize=11.8 K 2023-04-12 15:56:13,324 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=60.38 KB/61830 for 4870003209f366bc50ef2084e6e2cf03 in 477ms, sequenceid=198, compaction requested=false 2023-04-12 15:56:13,325 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4870003209f366bc50ef2084e6e2cf03: 2023-04-12 15:56:13,325 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:13,325 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:56:13,325 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:56:13,325 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:13,325 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:56:13,325 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:56:13,326 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:13,326 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:13,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:13,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:13,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:13,327 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:13,327 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:56:13,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:13,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:13,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:13,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:13,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:13,328 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:56:13,329 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@133d7c7[Count = 0] remaining members to acquire global barrier 2023-04-12 15:56:13,329 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:56:13,329 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:13,330 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:13,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:13,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:13,331 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:56:13,331 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:56:13,331 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:13,331 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:56:13,331 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:56:13,332 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:56:13,332 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:13,332 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:56:13,332 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:13,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:13,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:13,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:13,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:13,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:13,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:13,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:13,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:13,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:13,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:13,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:56:13,335 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:56:13,335 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:56:13,335 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:56:13,336 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:56:13,336 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:56:13,336 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:56:13,337 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:13,337 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:13,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:13,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:13,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:13,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:13,337 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:13,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:13,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:13,338 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:13,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:13,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:13,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:13,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:13,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:13,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:13,340 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:13,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:13,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:13,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:13,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:13,342 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:13,342 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:13,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:56:13,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:56:13,342 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:13,342 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:13,342 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:13,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:13,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:13,342 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:13,343 DEBUG [Thread-4110] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:56:13,342 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:13,343 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:13,343 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:13,343 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:13,343 DEBUG [Thread-4110] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:56:13,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:13,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:13,343 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:56:13,443 DEBUG [Thread-4110] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:56:13,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:56:13,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:56:13,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:56:13,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:56:13,446 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:56:13,446 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:13,446 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:56:13,446 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:56:13,447 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:13,447 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:13,452 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:13,452 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:13,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:13,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:13,453 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:13,453 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:56:13,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:13,453 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:13,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:56:13,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:13,454 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:56:13,454 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:56:13,454 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:56:13,454 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:56:13,454 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:13,454 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:56:13,454 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:56:13,454 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:56:13,454 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:13,455 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. started... 2023-04-12 15:56:13,455 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4870003209f366bc50ef2084e6e2cf03 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-12 15:56:13,455 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=A 2023-04-12 15:56:13,455 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:13,455 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=B 2023-04-12 15:56:13,456 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:13,456 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=C 2023-04-12 15:56:13,456 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:13,468 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041227ea9efb2742478a8a5bd33564157674 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041227ea9efb2742478a8a5bd33564157674 2023-04-12 15:56:13,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on 4870003209f366bc50ef2084e6e2cf03 2023-04-12 15:56:13,538 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. as already flushing 2023-04-12 15:56:13,559 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:13,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53360 deadline: 1681315033556, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:13,559 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:13,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53392 deadline: 1681315033557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:13,561 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:13,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53324 deadline: 1681315033559, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:13,661 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:13,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53360 deadline: 1681315033660, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:13,662 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:13,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53392 deadline: 1681315033660, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:13,663 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:13,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53324 deadline: 1681315033663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:13,865 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:13,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53392 deadline: 1681315033863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:13,867 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:13,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53360 deadline: 1681315033864, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:13,867 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:13,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53324 deadline: 1681315033865, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:13,873 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=213, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/A/dc730c770c004e8ea20bb161b82e625d 2023-04-12 15:56:13,883 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=213 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/B/f0886de0f31943689c0fc9154a0922d2 2023-04-12 15:56:14,151 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:14,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53292 deadline: 1681315034150, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:14,153 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:14,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53306 deadline: 1681315034152, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:14,168 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:14,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53392 deadline: 1681315034167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:14,172 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:14,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53360 deadline: 1681315034170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:14,172 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:14,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53324 deadline: 1681315034170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:14,318 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=213 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/C/dc8c7268701c45fc8129e59e4aa97585 2023-04-12 15:56:14,321 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/A/dc730c770c004e8ea20bb161b82e625d as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/dc730c770c004e8ea20bb161b82e625d 2023-04-12 15:56:14,325 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/dc730c770c004e8ea20bb161b82e625d, entries=150, sequenceid=213, filesize=25.3 K 2023-04-12 15:56:14,325 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/B/f0886de0f31943689c0fc9154a0922d2 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/f0886de0f31943689c0fc9154a0922d2 2023-04-12 15:56:14,334 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/f0886de0f31943689c0fc9154a0922d2, entries=150, sequenceid=213, filesize=11.8 K 2023-04-12 15:56:14,335 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/C/dc8c7268701c45fc8129e59e4aa97585 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/dc8c7268701c45fc8129e59e4aa97585 2023-04-12 15:56:14,350 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/dc8c7268701c45fc8129e59e4aa97585, entries=150, sequenceid=213, filesize=11.8 K 2023-04-12 15:56:14,350 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 4870003209f366bc50ef2084e6e2cf03 in 895ms, sequenceid=213, compaction requested=true 2023-04-12 15:56:14,350 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4870003209f366bc50ef2084e6e2cf03: 2023-04-12 15:56:14,351 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:14,351 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:56:14,351 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:56:14,351 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:14,351 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:56:14,351 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:56:14,352 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:14,352 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:14,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:14,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:14,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:14,353 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:14,353 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:56:14,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:14,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:14,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:14,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:14,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:14,355 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:56:14,355 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f410577[Count = 0] remaining members to acquire global barrier 2023-04-12 15:56:14,355 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:56:14,355 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:14,357 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:14,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:14,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:14,357 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:56:14,357 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:56:14,357 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:56:14,357 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:14,357 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:56:14,362 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:56:14,362 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:14,362 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:14,362 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:56:14,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:14,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:14,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:14,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:14,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:14,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:14,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:14,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:14,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:14,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:14,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:56:14,365 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:56:14,365 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:56:14,365 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:56:14,365 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:56:14,365 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:56:14,365 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:56:14,366 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:14,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:14,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:14,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:14,366 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:14,366 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:14,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:14,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:14,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:14,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:14,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:14,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:14,368 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:14,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:14,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:14,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:14,369 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:14,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:14,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:14,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:14,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:14,371 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:14,371 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:14,371 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:14,371 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:14,371 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:14,371 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:14,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:14,371 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:56:14,371 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:14,371 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:14,371 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:14,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:56:14,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:56:14,372 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:14,372 DEBUG [Thread-4110] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:56:14,372 DEBUG [Thread-4110] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:56:14,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:14,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:14,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:14,472 DEBUG [Thread-4110] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:56:14,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:56:14,474 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:56:14,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:56:14,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:56:14,475 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:56:14,475 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:14,475 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:56:14,475 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:56:14,476 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:14,476 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:14,477 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:14,477 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:14,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:14,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:14,478 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:14,478 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:56:14,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:14,478 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:14,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:56:14,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:14,478 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:56:14,479 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:56:14,479 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:56:14,479 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:56:14,479 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:14,479 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:56:14,479 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:56:14,479 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:56:14,479 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:14,479 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. started... 2023-04-12 15:56:14,479 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4870003209f366bc50ef2084e6e2cf03 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-12 15:56:14,479 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=A 2023-04-12 15:56:14,479 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:14,480 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=B 2023-04-12 15:56:14,480 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:14,480 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=C 2023-04-12 15:56:14,480 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:14,493 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304123d9f7b99d3304ff9a414ab308da6244e to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304123d9f7b99d3304ff9a414ab308da6244e 2023-04-12 15:56:14,498 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=233, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/A/99515b6745ed440f80a0ad2bf9f8481c 2023-04-12 15:56:14,508 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=233 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/B/e3bc615cc2ad43d1aeabd0924dbfc509 2023-04-12 15:56:14,517 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=233 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/C/9822f06bd891491786541d15b386f2d0 2023-04-12 15:56:14,521 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/A/99515b6745ed440f80a0ad2bf9f8481c as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/99515b6745ed440f80a0ad2bf9f8481c 2023-04-12 15:56:14,525 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/99515b6745ed440f80a0ad2bf9f8481c, entries=150, sequenceid=233, filesize=25.3 K 2023-04-12 15:56:14,527 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/B/e3bc615cc2ad43d1aeabd0924dbfc509 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/e3bc615cc2ad43d1aeabd0924dbfc509 2023-04-12 15:56:14,533 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/e3bc615cc2ad43d1aeabd0924dbfc509, entries=150, sequenceid=233, filesize=11.8 K 2023-04-12 15:56:14,544 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/C/9822f06bd891491786541d15b386f2d0 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/9822f06bd891491786541d15b386f2d0 2023-04-12 15:56:14,557 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/9822f06bd891491786541d15b386f2d0, entries=150, sequenceid=233, filesize=11.8 K 2023-04-12 15:56:14,558 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=0 B/0 for 4870003209f366bc50ef2084e6e2cf03 in 79ms, sequenceid=233, compaction requested=true 2023-04-12 15:56:14,558 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4870003209f366bc50ef2084e6e2cf03: 2023-04-12 15:56:14,558 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:14,558 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:56:14,558 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:56:14,558 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:14,559 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:56:14,559 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:56:14,566 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:14,566 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:14,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:14,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:14,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:14,567 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:14,567 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:56:14,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:14,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:14,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:14,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:14,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:14,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:56:14,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37e1b408[Count = 0] remaining members to acquire global barrier 2023-04-12 15:56:14,569 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:56:14,569 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:14,570 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:14,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:14,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:14,571 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:56:14,571 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:56:14,571 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:56:14,571 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:14,571 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:56:14,572 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:56:14,572 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:14,572 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:14,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:14,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:14,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:14,572 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:56:14,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:14,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:14,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:14,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:14,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:14,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:14,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:14,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:56:14,575 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:56:14,575 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:56:14,575 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:56:14,575 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:56:14,575 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:56:14,575 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:56:14,577 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:14,577 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:14,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:14,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:14,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:14,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:14,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:14,577 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:14,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:14,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:14,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:14,577 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:14,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:14,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:14,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:14,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:14,579 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:14,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:14,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:14,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:14,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:14,582 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:14,583 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:14,583 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:14,583 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:14,583 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:14,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:56:14,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:56:14,583 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:14,583 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:14,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:14,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:14,583 DEBUG [Thread-4110] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:56:14,583 DEBUG [Thread-4110] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:56:14,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:14,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:14,583 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:14,584 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:14,583 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:14,683 DEBUG [Thread-4110] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:56:14,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:56:14,688 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:56:14,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:56:14,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:56:14,704 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:56:14,704 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:14,704 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:56:14,704 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:56:14,704 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:14,705 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:14,706 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:14,706 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:14,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:14,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:14,706 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:14,706 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:56:14,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:14,707 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:14,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:56:14,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:14,708 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:56:14,708 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:56:14,708 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:56:14,708 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:56:14,708 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:14,708 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:56:14,708 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:56:14,708 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:56:14,709 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:14,709 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. started... 2023-04-12 15:56:14,709 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4870003209f366bc50ef2084e6e2cf03 3/3 column families, dataSize=13.42 KB heapSize=40.59 KB 2023-04-12 15:56:14,716 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=A 2023-04-12 15:56:14,716 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:14,717 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=B 2023-04-12 15:56:14,717 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:14,717 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=C 2023-04-12 15:56:14,717 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:14,746 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. as already flushing 2023-04-12 15:56:14,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on 4870003209f366bc50ef2084e6e2cf03 2023-04-12 15:56:14,783 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681314974773, max timestamp=1681314974779Num uniques 859; 2023-04-12 15:56:14,783 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-12 15:56:14,783 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304125bad526d58d94a8e8137b6c6ea22ad57 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304125bad526d58d94a8e8137b6c6ea22ad57 2023-04-12 15:56:14,783 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681314974773, max timestamp=1681314974780Num uniques 859; 2023-04-12 15:56:14,783 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681314974773, max timestamp=1681314974780Num uniques 859; 2023-04-12 15:56:14,783 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-12 15:56:14,784 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-12 15:56:14,798 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=239, memsize=6.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/A/aa4267b177a14e5998333550f082d8a6 2023-04-12 15:56:14,825 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681314974813, max timestamp=1681314974823Num uniques 1018; 2023-04-12 15:56:14,826 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-12 15:56:14,826 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681314974813, max timestamp=1681314974823Num uniques 1018; 2023-04-12 15:56:14,826 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681314974813, max timestamp=1681314974823Num uniques 1018; 2023-04-12 15:56:14,826 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-12 15:56:14,826 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-12 15:56:14,846 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=239 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/B/afc67f0e441448358a3612f3d25e97ad 2023-04-12 15:56:14,870 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681314974859, max timestamp=1681314974868Num uniques 1027; 2023-04-12 15:56:14,870 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681314974859, max timestamp=1681314974868Num uniques 1027; 2023-04-12 15:56:14,871 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-12 15:56:14,871 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-12 15:56:14,871 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681314974859, max timestamp=1681314974868Num uniques 1027; 2023-04-12 15:56:14,871 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-12 15:56:14,871 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=239 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/C/45072bbdd9f5454a9dafd915d4100f68 2023-04-12 15:56:14,876 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/A/aa4267b177a14e5998333550f082d8a6 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/aa4267b177a14e5998333550f082d8a6 2023-04-12 15:56:14,882 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/aa4267b177a14e5998333550f082d8a6, entries=100, sequenceid=239, filesize=18.5 K 2023-04-12 15:56:14,882 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/B/afc67f0e441448358a3612f3d25e97ad as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/afc67f0e441448358a3612f3d25e97ad 2023-04-12 15:56:14,887 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/afc67f0e441448358a3612f3d25e97ad, entries=100, sequenceid=239, filesize=9.4 K 2023-04-12 15:56:14,887 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/C/45072bbdd9f5454a9dafd915d4100f68 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/45072bbdd9f5454a9dafd915d4100f68 2023-04-12 15:56:14,893 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/45072bbdd9f5454a9dafd915d4100f68, entries=100, sequenceid=239, filesize=9.4 K 2023-04-12 15:56:14,894 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=107.34 KB/109920 for 4870003209f366bc50ef2084e6e2cf03 in 185ms, sequenceid=239, compaction requested=true 2023-04-12 15:56:14,894 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4870003209f366bc50ef2084e6e2cf03: 2023-04-12 15:56:14,894 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:14,894 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:56:14,894 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:56:14,894 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:14,894 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:56:14,894 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:56:14,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on 4870003209f366bc50ef2084e6e2cf03 2023-04-12 15:56:14,895 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4870003209f366bc50ef2084e6e2cf03 3/3 column families, dataSize=120.76 KB heapSize=343.45 KB 2023-04-12 15:56:14,895 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=A 2023-04-12 15:56:14,895 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:56:14,895 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=B 2023-04-12 15:56:14,896 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:56:14,896 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=C 2023-04-12 15:56:14,896 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:56:14,896 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:14,896 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:14,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:14,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:14,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:14,896 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:14,896 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:56:14,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:14,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:14,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:14,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:14,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:14,898 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:56:14,898 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:56:14,898 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:14,898 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2879e1b4[Count = 0] remaining members to acquire global barrier 2023-04-12 15:56:14,900 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:14,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:14,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:14,900 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:56:14,900 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:56:14,900 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:56:14,900 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:14,900 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:56:14,902 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:14,902 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:56:14,902 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:14,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:14,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:14,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:14,902 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:56:14,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:14,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:14,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:14,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:14,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:14,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:14,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:14,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:56:14,905 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:56:14,905 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:56:14,905 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:56:14,905 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:56:14,905 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:56:14,905 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:56:14,907 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:14,907 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:14,907 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:14,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:14,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:14,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:14,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:14,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:14,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:14,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:14,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:14,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:14,907 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:14,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:14,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:14,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:14,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:14,909 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:14,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:14,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:14,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:14,911 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:14,911 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:14,911 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:14,911 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:14,911 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:14,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:14,911 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:14,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:14,911 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:14,911 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:14,911 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:56:14,911 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:56:14,911 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:14,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:14,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:14,911 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:14,912 DEBUG [Thread-4110] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:56:14,912 DEBUG [Thread-4110] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:56:14,954 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:14,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53324 deadline: 1681315034949, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:14,955 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:14,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53392 deadline: 1681315034952, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:14,955 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:14,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53360 deadline: 1681315034952, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,012 DEBUG [Thread-4110] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:56:15,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:56:15,014 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:56:15,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:56:15,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:56:15,016 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:56:15,016 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:15,016 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:56:15,016 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:56:15,016 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:15,017 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:15,018 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:15,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:15,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:15,018 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:15,019 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,019 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:56:15,019 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:15,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:56:15,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:15,019 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:56:15,019 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:56:15,020 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:56:15,020 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:56:15,020 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:15,020 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:56:15,020 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:56:15,020 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:56:15,020 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:15,020 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. started... 2023-04-12 15:56:15,021 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. as already flushing 2023-04-12 15:56:15,021 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:15,021 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:56:15,021 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:56:15,021 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,021 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:56:15,021 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:56:15,023 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,023 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:15,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:15,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:15,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:15,024 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:15,024 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:56:15,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:15,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:15,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:15,025 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:56:15,025 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37eb8693[Count = 0] remaining members to acquire global barrier 2023-04-12 15:56:15,025 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:56:15,025 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:15,027 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:15,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:15,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:15,027 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:56:15,027 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:56:15,027 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:56:15,027 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,027 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:56:15,028 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,028 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:56:15,029 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:15,029 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:56:15,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:15,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:15,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:15,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:15,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:15,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:15,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:15,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:56:15,031 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:56:15,031 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:56:15,031 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:56:15,031 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:56:15,031 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:56:15,031 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:56:15,033 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:15,033 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:15,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:15,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:15,033 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:15,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:15,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:15,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:15,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:15,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:15,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:15,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:15,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:15,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:15,035 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:15,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:15,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:15,036 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,042 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:15,042 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,042 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:15,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:15,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:15,042 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:15,042 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:15,042 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:56:15,042 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:56:15,042 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,042 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:15,042 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:15,042 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:15,042 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:15,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:15,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:15,042 DEBUG [Thread-4110] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:56:15,043 DEBUG [Thread-4110] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:56:15,058 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:15,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53324 deadline: 1681315035055, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,058 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:15,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53392 deadline: 1681315035056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,058 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:15,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53360 deadline: 1681315035056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,143 DEBUG [Thread-4110] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:56:15,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:56:15,145 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:56:15,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:56:15,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:56:15,146 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:56:15,146 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:15,146 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:56:15,146 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:56:15,147 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:15,147 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:15,151 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:15,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:15,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:15,151 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:15,151 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,151 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:56:15,152 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:15,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:56:15,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:15,152 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:56:15,152 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:56:15,152 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:56:15,152 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:56:15,152 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:15,153 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:56:15,153 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:56:15,153 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:56:15,153 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:15,153 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. started... 2023-04-12 15:56:15,153 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. as already flushing 2023-04-12 15:56:15,153 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:15,153 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:56:15,153 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:56:15,153 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,153 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:56:15,153 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:56:15,155 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,155 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:15,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:15,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:15,155 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:15,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:15,155 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:56:15,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:15,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:15,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:15,157 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:56:15,157 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:56:15,157 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@407657a4[Count = 0] remaining members to acquire global barrier 2023-04-12 15:56:15,157 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:15,158 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:15,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:15,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:15,158 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:56:15,158 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:56:15,158 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:56:15,158 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,159 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:56:15,160 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:15,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:15,161 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:56:15,161 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:15,161 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:56:15,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:15,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:15,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:15,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:15,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:15,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:56:15,164 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:56:15,164 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:56:15,164 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:56:15,164 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:56:15,164 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:56:15,164 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:56:15,165 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:15,165 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:15,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:15,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:15,165 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:15,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:15,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:15,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:15,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:15,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:15,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:15,166 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:15,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:15,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:15,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:15,167 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:15,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:15,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,170 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,170 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:15,170 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:15,172 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:15,172 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:15,172 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,172 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:15,172 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:15,172 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:15,172 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:15,172 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:56:15,172 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:56:15,172 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:56:15,173 DEBUG [Thread-4110] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:56:15,173 DEBUG [Thread-4110] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:56:15,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:15,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:15,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:15,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:15,264 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:15,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53324 deadline: 1681315035260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,264 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:15,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53392 deadline: 1681315035260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,265 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:15,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53360 deadline: 1681315035261, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,273 DEBUG [Thread-4110] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:56:15,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:56:15,275 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:56:15,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:56:15,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:56:15,277 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:56:15,277 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:15,277 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:56:15,277 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:56:15,277 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:15,277 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:15,279 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:15,279 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:15,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:15,279 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,279 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:56:15,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:15,280 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:15,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:56:15,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:15,280 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:56:15,280 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:56:15,280 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:56:15,281 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:56:15,281 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:15,281 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:56:15,281 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:56:15,281 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:56:15,281 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:15,281 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. started... 2023-04-12 15:56:15,281 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. as already flushing 2023-04-12 15:56:15,281 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:15,281 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:56:15,281 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:56:15,281 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,281 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:56:15,281 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:56:15,284 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,284 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:15,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:15,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:15,284 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:15,284 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:56:15,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:15,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:15,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:15,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:15,286 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:56:15,286 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a8dfa56[Count = 0] remaining members to acquire global barrier 2023-04-12 15:56:15,286 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:56:15,286 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:15,287 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:15,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:15,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:15,287 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:56:15,287 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:56:15,287 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:56:15,287 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,288 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:56:15,288 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:15,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:15,289 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:56:15,289 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:15,289 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:56:15,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:15,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:15,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:15,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:15,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:15,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:56:15,291 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:56:15,291 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:56:15,291 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:56:15,292 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:56:15,292 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:56:15,292 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:56:15,294 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:15,294 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:15,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:15,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:15,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:15,294 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:15,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:15,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:15,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:15,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:15,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:15,294 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:15,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:15,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:15,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:15,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,296 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:15,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:15,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,304 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:15,304 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:15,304 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,304 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:15,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:15,304 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:56:15,304 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:56:15,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:15,304 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:56:15,304 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:15,304 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:15,304 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,304 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:15,304 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:15,305 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:15,304 DEBUG [Thread-4110] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:56:15,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:15,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:15,305 DEBUG [Thread-4110] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:56:15,325 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304127c804f500f494435ad4b3cc3996d2046 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304127c804f500f494435ad4b3cc3996d2046 2023-04-12 15:56:15,334 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=308, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/A/5780b690fd37459ea70ebb514b42b34a 2023-04-12 15:56:15,346 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=308 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/B/3bd9468b1233439ea95cd0f943c05f5f 2023-04-12 15:56:15,358 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=308 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/C/9e5470faaf4942bda2fd94afe540f099 2023-04-12 15:56:15,363 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/A/5780b690fd37459ea70ebb514b42b34a as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/5780b690fd37459ea70ebb514b42b34a 2023-04-12 15:56:15,367 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/5780b690fd37459ea70ebb514b42b34a, entries=250, sequenceid=308, filesize=39.2 K 2023-04-12 15:56:15,368 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/B/3bd9468b1233439ea95cd0f943c05f5f as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/3bd9468b1233439ea95cd0f943c05f5f 2023-04-12 15:56:15,372 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/3bd9468b1233439ea95cd0f943c05f5f, entries=150, sequenceid=308, filesize=11.9 K 2023-04-12 15:56:15,373 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/C/9e5470faaf4942bda2fd94afe540f099 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/9e5470faaf4942bda2fd94afe540f099 2023-04-12 15:56:15,376 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/9e5470faaf4942bda2fd94afe540f099, entries=150, sequenceid=308, filesize=11.9 K 2023-04-12 15:56:15,377 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~343.41 KB/351648, currentSize=67.09 KB/68700 for 4870003209f366bc50ef2084e6e2cf03 in 482ms, sequenceid=308, compaction requested=true 2023-04-12 15:56:15,377 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4870003209f366bc50ef2084e6e2cf03: 2023-04-12 15:56:15,377 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:56:15,377 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:56:15,377 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-12 15:56:15,377 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-12 15:56:15,377 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:56:15,379 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 162988 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-12 15:56:15,379 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1912): 4870003209f366bc50ef2084e6e2cf03/A is initiating minor compaction (all files) 2023-04-12 15:56:15,379 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4870003209f366bc50ef2084e6e2cf03/A in TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:15,379 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/6cf94b918db84e4da8056344fdb7def6, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/803c0ebb03de497597a4fb2603433b4c, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/dc730c770c004e8ea20bb161b82e625d, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/99515b6745ed440f80a0ad2bf9f8481c, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/aa4267b177a14e5998333550f082d8a6, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/5780b690fd37459ea70ebb514b42b34a] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp, totalSize=159.2 K 2023-04-12 15:56:15,379 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 70348 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-12 15:56:15,379 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1912): 4870003209f366bc50ef2084e6e2cf03/B is initiating minor compaction (all files) 2023-04-12 15:56:15,379 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4870003209f366bc50ef2084e6e2cf03/B in TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:15,379 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/b01faf10215543e89ab551370b386ccc, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/70215b26de4541da8cd467d0489e6a9b, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/f0886de0f31943689c0fc9154a0922d2, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/e3bc615cc2ad43d1aeabd0924dbfc509, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/afc67f0e441448358a3612f3d25e97ad, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/3bd9468b1233439ea95cd0f943c05f5f] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp, totalSize=68.7 K 2023-04-12 15:56:15,380 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 6cf94b918db84e4da8056344fdb7def6, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=163, earliestPutTs=1681314971865 2023-04-12 15:56:15,380 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting b01faf10215543e89ab551370b386ccc, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=163, earliestPutTs=1681314971865 2023-04-12 15:56:15,380 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 803c0ebb03de497597a4fb2603433b4c, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=198, earliestPutTs=1681314972014 2023-04-12 15:56:15,380 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 70215b26de4541da8cd467d0489e6a9b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=198, earliestPutTs=1681314972014 2023-04-12 15:56:15,381 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting dc730c770c004e8ea20bb161b82e625d, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1681314972920 2023-04-12 15:56:15,381 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting f0886de0f31943689c0fc9154a0922d2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1681314972920 2023-04-12 15:56:15,381 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 99515b6745ed440f80a0ad2bf9f8481c, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=233, earliestPutTs=1681314973554 2023-04-12 15:56:15,381 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting e3bc615cc2ad43d1aeabd0924dbfc509, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=233, earliestPutTs=1681314973554 2023-04-12 15:56:15,381 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting aa4267b177a14e5998333550f082d8a6, keycount=100, bloomtype=ROW, size=18.5 K, encoding=NONE, compression=NONE, seqNum=239, earliestPutTs=1681314974703 2023-04-12 15:56:15,382 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting afc67f0e441448358a3612f3d25e97ad, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=239, earliestPutTs=1681314974703 2023-04-12 15:56:15,382 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 5780b690fd37459ea70ebb514b42b34a, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=308, earliestPutTs=1681314974883 2023-04-12 15:56:15,382 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 3bd9468b1233439ea95cd0f943c05f5f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=308, earliestPutTs=1681314974888 2023-04-12 15:56:15,401 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] throttle.PressureAwareThroughputController(145): 4870003209f366bc50ef2084e6e2cf03#B#compaction#605 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:56:15,405 DEBUG [Thread-4110] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:56:15,405 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4870003209f366bc50ef2084e6e2cf03#A#compaction#604 average throughput is 2.19 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:56:15,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:56:15,408 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:56:15,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:56:15,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:56:15,412 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:56:15,412 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:15,412 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:56:15,412 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:56:15,412 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:15,413 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:15,414 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,414 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:15,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:15,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:15,415 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,415 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:56:15,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:15,415 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:15,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:56:15,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:15,415 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:56:15,432 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:56:15,432 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:56:15,432 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:56:15,432 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:15,432 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:56:15,440 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:56:15,440 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:56:15,440 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:15,440 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. started... 2023-04-12 15:56:15,440 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4870003209f366bc50ef2084e6e2cf03 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-12 15:56:15,440 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=A 2023-04-12 15:56:15,440 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:15,440 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=B 2023-04-12 15:56:15,441 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:15,441 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=C 2023-04-12 15:56:15,441 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:15,465 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304125f7f4cbb7f184a0f854a8ace437430af to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304125f7f4cbb7f184a0f854a8ace437430af 2023-04-12 15:56:15,495 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=321, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/A/aae955ad339f4e63b09c8ac9863f761a 2023-04-12 15:56:15,531 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=321 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/B/a89230cf08c442de9c60a355b7341823 2023-04-12 15:56:15,560 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=321 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/C/d848244d3cee450799a65fbdae5ac99f 2023-04-12 15:56:15,564 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/A/aae955ad339f4e63b09c8ac9863f761a as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/aae955ad339f4e63b09c8ac9863f761a 2023-04-12 15:56:15,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on 4870003209f366bc50ef2084e6e2cf03 2023-04-12 15:56:15,569 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. as already flushing 2023-04-12 15:56:15,575 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/aae955ad339f4e63b09c8ac9863f761a, entries=100, sequenceid=321, filesize=18.6 K 2023-04-12 15:56:15,576 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/B/a89230cf08c442de9c60a355b7341823 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/a89230cf08c442de9c60a355b7341823 2023-04-12 15:56:15,583 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/a89230cf08c442de9c60a355b7341823, entries=100, sequenceid=321, filesize=9.5 K 2023-04-12 15:56:15,584 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/C/d848244d3cee450799a65fbdae5ac99f as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/d848244d3cee450799a65fbdae5ac99f 2023-04-12 15:56:15,597 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/d848244d3cee450799a65fbdae5ac99f, entries=100, sequenceid=321, filesize=9.5 K 2023-04-12 15:56:15,597 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=67.09 KB/68700 for 4870003209f366bc50ef2084e6e2cf03 in 157ms, sequenceid=321, compaction requested=true 2023-04-12 15:56:15,598 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4870003209f366bc50ef2084e6e2cf03: 2023-04-12 15:56:15,598 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:15,598 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:56:15,598 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:56:15,598 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,598 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:56:15,598 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:56:15,606 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,606 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:15,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:15,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:15,606 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:15,607 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:56:15,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:15,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:15,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:15,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:15,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:56:15,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d1ec0f2[Count = 0] remaining members to acquire global barrier 2023-04-12 15:56:15,609 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:56:15,609 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:15,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on 4870003209f366bc50ef2084e6e2cf03 2023-04-12 15:56:15,610 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4870003209f366bc50ef2084e6e2cf03 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-12 15:56:15,610 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=A 2023-04-12 15:56:15,610 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:15,610 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=B 2023-04-12 15:56:15,610 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:15,610 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=C 2023-04-12 15:56:15,610 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:15,611 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:15,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:15,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:15,612 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:56:15,612 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:56:15,612 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:56:15,612 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,613 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:56:15,615 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:56:15,615 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:15,615 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:56:15,615 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:15,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:15,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:15,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:15,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:15,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:15,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:15,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:56:15,619 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:56:15,619 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:56:15,619 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:56:15,619 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:56:15,619 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:56:15,619 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:56:15,621 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:15,621 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:15,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:15,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:15,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:15,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:15,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:15,624 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:15,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:15,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:15,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:15,625 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:15,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:15,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:15,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:15,627 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:15,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:15,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,630 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:15,630 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:15,630 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,630 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:15,630 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:15,630 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,630 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:15,630 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:15,630 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:15,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:15,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:15,630 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:15,630 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:56:15,630 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:56:15,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:15,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:15,630 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:56:15,631 DEBUG [Thread-4110] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:56:15,631 DEBUG [Thread-4110] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:56:15,654 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041256ae6ceaf48b4c94a9801898356c7bb4 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041256ae6ceaf48b4c94a9801898356c7bb4 2023-04-12 15:56:15,666 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:15,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53392 deadline: 1681315035659, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,666 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:15,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53360 deadline: 1681315035661, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,667 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:15,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53324 deadline: 1681315035662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,676 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=335, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/A/8a288ec7dcee4b68b0dcb6153e0dc7f2 2023-04-12 15:56:15,714 WARN [DataStreamer for file /user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/B/3745e854d90d484dbbdd945b3a0f36fb block BP-1985789108-172.31.14.131-1681314814180:blk_1073742557_1733] hdfs.DataStreamer(982): Caught exception java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1257) at java.lang.Thread.join(Thread.java:1331) at org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:980) at org.apache.hadoop.hdfs.DataStreamer.closeInternal(DataStreamer.java:844) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:840) 2023-04-12 15:56:15,717 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=335 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/B/3745e854d90d484dbbdd945b3a0f36fb 2023-04-12 15:56:15,731 DEBUG [Thread-4110] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:56:15,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:56:15,734 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:56:15,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:56:15,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:56:15,745 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:56:15,745 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:15,745 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:56:15,745 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:56:15,746 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:15,746 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:15,748 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:15,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:15,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:15,748 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,749 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,749 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:56:15,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:15,749 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:15,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:56:15,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:15,750 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:56:15,750 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:56:15,750 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:56:15,750 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:56:15,750 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:15,750 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:56:15,756 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:56:15,756 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:56:15,756 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=335 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/C/45ecf715677543ecb84bc04fd8ffe1bb 2023-04-12 15:56:15,757 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:15,757 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. started... 2023-04-12 15:56:15,760 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. as already flushing 2023-04-12 15:56:15,760 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:15,760 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:56:15,760 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:56:15,760 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,760 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:56:15,760 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:56:15,762 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:15,762 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:15,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:15,762 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:15,762 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:56:15,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:15,763 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/A/8a288ec7dcee4b68b0dcb6153e0dc7f2 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/8a288ec7dcee4b68b0dcb6153e0dc7f2 2023-04-12 15:56:15,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:15,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:15,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:15,767 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:56:15,767 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76aa12dc[Count = 0] remaining members to acquire global barrier 2023-04-12 15:56:15,767 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:56:15,767 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:15,769 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:15,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:15,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:15,769 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:56:15,769 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:56:15,769 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,769 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:56:15,769 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:56:15,770 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:56:15,770 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:15,770 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,770 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:56:15,771 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/8a288ec7dcee4b68b0dcb6153e0dc7f2, entries=300, sequenceid=335, filesize=46.1 K 2023-04-12 15:56:15,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:15,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:15,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:15,772 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/B/3745e854d90d484dbbdd945b3a0f36fb as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/3745e854d90d484dbbdd945b3a0f36fb 2023-04-12 15:56:15,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:15,773 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:15,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53392 deadline: 1681315035769, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:15,773 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:15,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53324 deadline: 1681315035769, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:15,774 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:15,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53360 deadline: 1681315035769, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:15,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:56:15,776 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:56:15,776 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:56:15,776 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:56:15,776 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:56:15,776 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:56:15,776 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:56:15,778 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/3745e854d90d484dbbdd945b3a0f36fb, entries=100, sequenceid=335, filesize=9.5 K 2023-04-12 15:56:15,779 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/C/45ecf715677543ecb84bc04fd8ffe1bb as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/45ecf715677543ecb84bc04fd8ffe1bb 2023-04-12 15:56:15,780 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:15,780 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:15,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:15,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:15,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:15,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:15,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:15,780 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:15,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:15,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:15,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:15,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:15,782 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:15,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:15,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:15,783 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:15,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:15,784 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/45ecf715677543ecb84bc04fd8ffe1bb, entries=100, sequenceid=335, filesize=9.5 K 2023-04-12 15:56:15,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,785 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 4870003209f366bc50ef2084e6e2cf03 in 175ms, sequenceid=335, compaction requested=true 2023-04-12 15:56:15,785 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4870003209f366bc50ef2084e6e2cf03: 2023-04-12 15:56:15,785 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-12 15:56:15,785 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-04-12 15:56:15,785 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=4), splitQueue=0 2023-04-12 15:56:15,786 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,786 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:15,786 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:15,786 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:15,786 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:15,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:15,786 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:56:15,786 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:56:15,786 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:15,786 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:15,787 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:56:15,787 DEBUG [Thread-4110] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:56:15,787 DEBUG [Thread-4110] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:56:15,786 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:15,787 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:15,787 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:15,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:15,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:15,835 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/B/6acefc2ba0b5407eacdcf759ecde9c21 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/6acefc2ba0b5407eacdcf759ecde9c21 2023-04-12 15:56:15,843 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 4870003209f366bc50ef2084e6e2cf03/B of 4870003209f366bc50ef2084e6e2cf03 into 6acefc2ba0b5407eacdcf759ecde9c21(size=12.5 K), total size for store is 31.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:56:15,843 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4870003209f366bc50ef2084e6e2cf03: 2023-04-12 15:56:15,843 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03., storeName=4870003209f366bc50ef2084e6e2cf03/B, priority=10, startTime=1681314975377; duration=0sec 2023-04-12 15:56:15,843 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=4), splitQueue=0 2023-04-12 15:56:15,843 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 6 compacting, 2 eligible, 16 blocking 2023-04-12 15:56:15,844 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-12 15:56:15,844 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-12 15:56:15,845 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. because compaction request was cancelled 2023-04-12 15:56:15,845 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-04-12 15:56:15,846 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 89832 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-04-12 15:56:15,847 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1912): 4870003209f366bc50ef2084e6e2cf03/C is initiating minor compaction (all files) 2023-04-12 15:56:15,847 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4870003209f366bc50ef2084e6e2cf03/C in TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:15,847 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/d76c5cda3a31405997aa8220567caf75, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/96a6307aae664f5a9d1863cdbddd6080, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/dc8c7268701c45fc8129e59e4aa97585, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/9822f06bd891491786541d15b386f2d0, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/45072bbdd9f5454a9dafd915d4100f68, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/9e5470faaf4942bda2fd94afe540f099, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/d848244d3cee450799a65fbdae5ac99f, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/45ecf715677543ecb84bc04fd8ffe1bb] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp, totalSize=87.7 K 2023-04-12 15:56:15,847 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting d76c5cda3a31405997aa8220567caf75, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=163, earliestPutTs=1681314971865 2023-04-12 15:56:15,847 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 96a6307aae664f5a9d1863cdbddd6080, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=198, earliestPutTs=1681314972014 2023-04-12 15:56:15,848 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting dc8c7268701c45fc8129e59e4aa97585, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1681314972920 2023-04-12 15:56:15,848 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 9822f06bd891491786541d15b386f2d0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=233, earliestPutTs=1681314973554 2023-04-12 15:56:15,848 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 45072bbdd9f5454a9dafd915d4100f68, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=239, earliestPutTs=1681314974703 2023-04-12 15:56:15,849 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 9e5470faaf4942bda2fd94afe540f099, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=308, earliestPutTs=1681314974888 2023-04-12 15:56:15,849 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting d848244d3cee450799a65fbdae5ac99f, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=321, earliestPutTs=1681314974950 2023-04-12 15:56:15,849 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 45ecf715677543ecb84bc04fd8ffe1bb, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=335, earliestPutTs=1681314975589 2023-04-12 15:56:15,871 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] throttle.PressureAwareThroughputController(145): 4870003209f366bc50ef2084e6e2cf03#C#compaction#612 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:56:15,887 DEBUG [Thread-4110] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:56:15,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:56:15,890 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:56:15,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:56:15,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:56:15,901 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:56:15,901 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:15,901 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:56:15,901 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:56:15,902 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:15,902 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:15,903 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:15,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:15,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:15,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:15,904 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,905 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:15,905 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:15,905 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:56:15,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:56:15,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:15,905 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:56:15,928 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:56:15,928 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:56:15,928 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:56:15,928 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:15,928 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:56:15,932 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:56:15,932 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:56:15,932 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:15,932 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. started... 2023-04-12 15:56:15,933 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4870003209f366bc50ef2084e6e2cf03 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-12 15:56:15,933 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=A 2023-04-12 15:56:15,933 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:15,933 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=B 2023-04-12 15:56:15,933 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:15,933 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=C 2023-04-12 15:56:15,933 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:15,938 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/C/b3c93ab9cbc3460cb52f0908ee68cfd2 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/b3c93ab9cbc3460cb52f0908ee68cfd2 2023-04-12 15:56:15,944 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in 4870003209f366bc50ef2084e6e2cf03/C of 4870003209f366bc50ef2084e6e2cf03 into b3c93ab9cbc3460cb52f0908ee68cfd2(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:56:15,944 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4870003209f366bc50ef2084e6e2cf03: 2023-04-12 15:56:15,945 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03., storeName=4870003209f366bc50ef2084e6e2cf03/C, priority=8, startTime=1681314975785; duration=0sec 2023-04-12 15:56:15,945 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-12 15:56:15,945 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 15:56:15,946 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 32250 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 15:56:15,946 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1912): 4870003209f366bc50ef2084e6e2cf03/B is initiating minor compaction (all files) 2023-04-12 15:56:15,946 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4870003209f366bc50ef2084e6e2cf03/B in TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:15,946 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/6acefc2ba0b5407eacdcf759ecde9c21, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/a89230cf08c442de9c60a355b7341823, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/3745e854d90d484dbbdd945b3a0f36fb] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp, totalSize=31.5 K 2023-04-12 15:56:15,946 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 6acefc2ba0b5407eacdcf759ecde9c21, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=308, earliestPutTs=1681314974888 2023-04-12 15:56:15,947 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting a89230cf08c442de9c60a355b7341823, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=321, earliestPutTs=1681314974950 2023-04-12 15:56:15,947 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 3745e854d90d484dbbdd945b3a0f36fb, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=335, earliestPutTs=1681314975589 2023-04-12 15:56:15,957 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] throttle.PressureAwareThroughputController(145): 4870003209f366bc50ef2084e6e2cf03#B#compaction#613 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:56:15,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on 4870003209f366bc50ef2084e6e2cf03 2023-04-12 15:56:15,978 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. as already flushing 2023-04-12 15:56:15,985 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/A/b2bdb591e16e427dbe6db6674ffeae25 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/b2bdb591e16e427dbe6db6674ffeae25 2023-04-12 15:56:15,991 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 4870003209f366bc50ef2084e6e2cf03/A of 4870003209f366bc50ef2084e6e2cf03 into b2bdb591e16e427dbe6db6674ffeae25(size=26.0 K), total size for store is 90.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:56:15,992 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4870003209f366bc50ef2084e6e2cf03: 2023-04-12 15:56:15,992 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03., storeName=4870003209f366bc50ef2084e6e2cf03/A, priority=10, startTime=1681314975377; duration=0sec 2023-04-12 15:56:15,992 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:56:15,992 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 1 store files, 0 compacting, 1 eligible, 16 blocking 2023-04-12 15:56:15,992 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-12 15:56:15,993 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-12 15:56:15,993 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. because compaction request was cancelled 2023-04-12 15:56:15,999 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/B/15ed22be7f464fa59581fe57b17f67ec as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/15ed22be7f464fa59581fe57b17f67ec 2023-04-12 15:56:16,000 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:16,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53360 deadline: 1681315035995, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:16,001 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:16,001 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230412ba3d370591de4a8183dea2e2144292f3 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412ba3d370591de4a8183dea2e2144292f3 2023-04-12 15:56:16,001 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:16,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53392 deadline: 1681315035995, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:16,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53324 deadline: 1681315035996, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:16,004 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4870003209f366bc50ef2084e6e2cf03/B of 4870003209f366bc50ef2084e6e2cf03 into 15ed22be7f464fa59581fe57b17f67ec(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:56:16,004 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4870003209f366bc50ef2084e6e2cf03: 2023-04-12 15:56:16,004 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03., storeName=4870003209f366bc50ef2084e6e2cf03/B, priority=13, startTime=1681314975785; duration=0sec 2023-04-12 15:56:16,004 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:56:16,020 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=354, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/A/642acdba67f84af2b20066723551fa47 2023-04-12 15:56:16,038 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=354 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/B/d260f3cb759f418ea7cfb00737f0d53c 2023-04-12 15:56:16,050 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=354 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/C/8c1108d3fcde40528b3ce3526089f927 2023-04-12 15:56:16,053 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/A/642acdba67f84af2b20066723551fa47 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/642acdba67f84af2b20066723551fa47 2023-04-12 15:56:16,057 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/642acdba67f84af2b20066723551fa47, entries=150, sequenceid=354, filesize=25.4 K 2023-04-12 15:56:16,058 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/B/d260f3cb759f418ea7cfb00737f0d53c as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/d260f3cb759f418ea7cfb00737f0d53c 2023-04-12 15:56:16,061 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/d260f3cb759f418ea7cfb00737f0d53c, entries=150, sequenceid=354, filesize=11.9 K 2023-04-12 15:56:16,062 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/C/8c1108d3fcde40528b3ce3526089f927 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/8c1108d3fcde40528b3ce3526089f927 2023-04-12 15:56:16,065 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/8c1108d3fcde40528b3ce3526089f927, entries=150, sequenceid=354, filesize=11.9 K 2023-04-12 15:56:16,066 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 4870003209f366bc50ef2084e6e2cf03 in 133ms, sequenceid=354, compaction requested=true 2023-04-12 15:56:16,066 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4870003209f366bc50ef2084e6e2cf03: 2023-04-12 15:56:16,066 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:16,066 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:56:16,066 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:56:16,066 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:16,067 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:56:16,067 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:56:16,068 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:16,068 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:16,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:16,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:16,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:16,068 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:16,068 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:56:16,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:16,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:16,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:16,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:16,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:16,070 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:56:16,070 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@413b0da[Count = 0] remaining members to acquire global barrier 2023-04-12 15:56:16,070 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:56:16,070 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:16,072 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:16,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:16,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:16,072 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:56:16,072 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:56:16,072 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:56:16,072 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:16,072 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:56:16,073 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:16,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:16,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:16,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:16,073 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:56:16,074 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:16,074 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:56:16,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:16,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:16,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:16,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:16,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:16,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:16,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:16,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:56:16,076 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:56:16,076 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:56:16,076 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:56:16,076 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:56:16,076 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:56:16,076 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:56:16,079 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:16,079 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:16,079 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:16,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:16,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:16,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:16,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:16,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:16,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:16,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:16,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:16,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:16,080 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:16,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:16,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:16,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:16,082 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:16,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:16,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:16,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:16,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:16,084 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:16,084 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:16,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:16,084 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:16,084 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:16,084 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:16,084 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:16,084 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:16,084 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:16,084 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:16,084 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:16,084 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:56:16,085 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:56:16,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:16,085 DEBUG [Thread-4110] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:56:16,085 DEBUG [Thread-4110] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:56:16,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:16,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:16,085 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:56:16,104 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4870003209f366bc50ef2084e6e2cf03 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-12 15:56:16,104 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=A 2023-04-12 15:56:16,104 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:16,104 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=B 2023-04-12 15:56:16,104 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:16,104 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=C 2023-04-12 15:56:16,104 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:16,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on 4870003209f366bc50ef2084e6e2cf03 2023-04-12 15:56:16,129 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:16,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53392 deadline: 1681315036125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:16,129 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:16,129 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:16,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53360 deadline: 1681315036125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:16,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53324 deadline: 1681315036126, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:16,149 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230412bd1060033d7a4df282b692afdd1a56b2 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412bd1060033d7a4df282b692afdd1a56b2 2023-04-12 15:56:16,162 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:16,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53292 deadline: 1681315036160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:16,169 DEBUG [Thread-4100] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4145 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03., hostname=jenkins-hbase4.apache.org,38905,1681314818739, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-12 15:56:16,171 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:16,171 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=372, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/A/1134d581aef84f2f8bd11b732ccbe78b 2023-04-12 15:56:16,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53306 deadline: 1681315036170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:16,175 DEBUG [Thread-4108] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4153 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03., hostname=jenkins-hbase4.apache.org,38905,1681314818739, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-12 15:56:16,185 DEBUG [Thread-4110] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:56:16,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:56:16,186 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=372 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/B/1734884edf2445bc8dac36d4f297a93a 2023-04-12 15:56:16,187 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:56:16,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:56:16,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:56:16,189 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:56:16,189 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:16,189 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:56:16,189 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:56:16,190 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:16,190 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:16,191 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:16,191 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:16,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:16,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:16,191 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:16,191 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:56:16,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:16,192 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:16,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:56:16,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:16,192 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:56:16,193 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:56:16,193 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:56:16,193 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:56:16,193 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:16,193 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:56:16,201 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:56:16,201 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:56:16,201 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:16,201 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. started... 2023-04-12 15:56:16,201 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. as already flushing 2023-04-12 15:56:16,201 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:16,201 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:56:16,201 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:56:16,201 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:16,201 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:56:16,201 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:56:16,205 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:16,205 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:16,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:16,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:16,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:16,205 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:16,205 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:56:16,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:16,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:16,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:16,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:16,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:16,210 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:56:16,210 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=372 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/C/f6344aa7a3514ab7b675fa186c0a6a91 2023-04-12 15:56:16,210 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:56:16,210 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:16,210 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1159b544[Count = 0] remaining members to acquire global barrier 2023-04-12 15:56:16,212 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:16,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:16,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:16,212 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:56:16,212 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:56:16,212 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:56:16,213 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:16,213 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:56:16,215 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:16,215 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:56:16,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:16,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:16,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:16,215 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:16,215 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:56:16,215 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/A/1134d581aef84f2f8bd11b732ccbe78b as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/1134d581aef84f2f8bd11b732ccbe78b 2023-04-12 15:56:16,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:16,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:16,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:16,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:16,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:16,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:16,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:16,219 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/1134d581aef84f2f8bd11b732ccbe78b, entries=200, sequenceid=372, filesize=32.3 K 2023-04-12 15:56:16,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:56:16,220 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:56:16,220 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:56:16,220 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:56:16,220 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:56:16,220 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:56:16,220 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:56:16,220 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/B/1734884edf2445bc8dac36d4f297a93a as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/1734884edf2445bc8dac36d4f297a93a 2023-04-12 15:56:16,222 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:16,222 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:16,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:16,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:16,222 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:16,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:16,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:16,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:16,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:16,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:16,222 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:16,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:16,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:16,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:16,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:16,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:16,224 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:16,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:16,225 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/1734884edf2445bc8dac36d4f297a93a, entries=150, sequenceid=372, filesize=11.9 K 2023-04-12 15:56:16,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:16,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:16,226 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/C/f6344aa7a3514ab7b675fa186c0a6a91 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/f6344aa7a3514ab7b675fa186c0a6a91 2023-04-12 15:56:16,227 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:16,227 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:16,227 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:16,227 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:56:16,227 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:56:16,227 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:16,227 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:16,227 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:16,227 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:16,227 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:16,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:16,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:16,227 DEBUG [Thread-4110] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:56:16,227 DEBUG [Thread-4110] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:56:16,227 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:16,227 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:16,228 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:56:16,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:16,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:16,229 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/f6344aa7a3514ab7b675fa186c0a6a91, entries=150, sequenceid=372, filesize=11.9 K 2023-04-12 15:56:16,230 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=100.63 KB/103050 for 4870003209f366bc50ef2084e6e2cf03 in 126ms, sequenceid=372, compaction requested=true 2023-04-12 15:56:16,230 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4870003209f366bc50ef2084e6e2cf03: 2023-04-12 15:56:16,231 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:56:16,231 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-12 15:56:16,231 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:56:16,231 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:56:16,231 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 15:56:16,232 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37240 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 15:56:16,232 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1912): 4870003209f366bc50ef2084e6e2cf03/B is initiating minor compaction (all files) 2023-04-12 15:56:16,232 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4870003209f366bc50ef2084e6e2cf03/B in TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:16,232 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/15ed22be7f464fa59581fe57b17f67ec, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/d260f3cb759f418ea7cfb00737f0d53c, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/1734884edf2445bc8dac36d4f297a93a] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp, totalSize=36.4 K 2023-04-12 15:56:16,232 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 151922 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-12 15:56:16,232 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1912): 4870003209f366bc50ef2084e6e2cf03/A is initiating minor compaction (all files) 2023-04-12 15:56:16,232 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4870003209f366bc50ef2084e6e2cf03/A in TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:16,232 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/b2bdb591e16e427dbe6db6674ffeae25, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/aae955ad339f4e63b09c8ac9863f761a, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/8a288ec7dcee4b68b0dcb6153e0dc7f2, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/642acdba67f84af2b20066723551fa47, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/1134d581aef84f2f8bd11b732ccbe78b] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp, totalSize=148.4 K 2023-04-12 15:56:16,232 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 15ed22be7f464fa59581fe57b17f67ec, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=335, earliestPutTs=1681314974950 2023-04-12 15:56:16,233 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting b2bdb591e16e427dbe6db6674ffeae25, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=308, earliestPutTs=1681314974888 2023-04-12 15:56:16,233 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting d260f3cb759f418ea7cfb00737f0d53c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=354, earliestPutTs=1681314975657 2023-04-12 15:56:16,233 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting aae955ad339f4e63b09c8ac9863f761a, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=321, earliestPutTs=1681314974950 2023-04-12 15:56:16,233 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 1734884edf2445bc8dac36d4f297a93a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=372, earliestPutTs=1681314975987 2023-04-12 15:56:16,233 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 8a288ec7dcee4b68b0dcb6153e0dc7f2, keycount=300, bloomtype=ROW, size=46.1 K, encoding=NONE, compression=NONE, seqNum=335, earliestPutTs=1681314975577 2023-04-12 15:56:16,234 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 642acdba67f84af2b20066723551fa47, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=354, earliestPutTs=1681314975657 2023-04-12 15:56:16,234 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 1134d581aef84f2f8bd11b732ccbe78b, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=372, earliestPutTs=1681314975987 2023-04-12 15:56:16,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on 4870003209f366bc50ef2084e6e2cf03 2023-04-12 15:56:16,236 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4870003209f366bc50ef2084e6e2cf03 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-12 15:56:16,237 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=A 2023-04-12 15:56:16,237 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:16,237 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=B 2023-04-12 15:56:16,237 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:16,237 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=C 2023-04-12 15:56:16,237 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:16,240 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] throttle.PressureAwareThroughputController(145): 4870003209f366bc50ef2084e6e2cf03#B#compaction#620 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:56:16,252 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:16,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53360 deadline: 1681315036249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:16,253 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:16,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53392 deadline: 1681315036249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:16,256 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:16,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53324 deadline: 1681315036252, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:16,276 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/B/d13099c165374393873d1dc2ca8dda2e as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/d13099c165374393873d1dc2ca8dda2e 2023-04-12 15:56:16,280 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4870003209f366bc50ef2084e6e2cf03/B of 4870003209f366bc50ef2084e6e2cf03 into d13099c165374393873d1dc2ca8dda2e(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:56:16,280 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4870003209f366bc50ef2084e6e2cf03: 2023-04-12 15:56:16,280 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03., storeName=4870003209f366bc50ef2084e6e2cf03/B, priority=13, startTime=1681314976231; duration=0sec 2023-04-12 15:56:16,280 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:56:16,280 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 15:56:16,281 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 15:56:16,281 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1912): 4870003209f366bc50ef2084e6e2cf03/C is initiating minor compaction (all files) 2023-04-12 15:56:16,282 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4870003209f366bc50ef2084e6e2cf03/C in TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:16,282 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/b3c93ab9cbc3460cb52f0908ee68cfd2, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/8c1108d3fcde40528b3ce3526089f927, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/f6344aa7a3514ab7b675fa186c0a6a91] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp, totalSize=36.3 K 2023-04-12 15:56:16,282 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4870003209f366bc50ef2084e6e2cf03#A#compaction#622 average throughput is 0.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:56:16,282 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting b3c93ab9cbc3460cb52f0908ee68cfd2, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=335, earliestPutTs=1681314974950 2023-04-12 15:56:16,283 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 8c1108d3fcde40528b3ce3526089f927, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=354, earliestPutTs=1681314975657 2023-04-12 15:56:16,283 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting f6344aa7a3514ab7b675fa186c0a6a91, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=372, earliestPutTs=1681314975987 2023-04-12 15:56:16,309 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304122c88d5c4ed214eaea4ce6d4c8d427cc0 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304122c88d5c4ed214eaea4ce6d4c8d427cc0 2023-04-12 15:56:16,328 DEBUG [Thread-4110] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:56:16,329 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] throttle.PressureAwareThroughputController(145): 4870003209f366bc50ef2084e6e2cf03#C#compaction#623 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:56:16,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:56:16,332 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:56:16,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:56:16,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:56:16,352 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:56:16,352 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:16,352 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:56:16,352 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:56:16,353 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:16,353 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:16,357 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:16,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53392 deadline: 1681315036354, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:16,357 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:16,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53360 deadline: 1681315036356, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:16,357 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=392, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/A/7a8f17bfe827445f96297fa297e562a8 2023-04-12 15:56:16,358 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:16,358 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:16,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:16,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:16,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:16,359 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:16,359 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:56:16,359 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:16,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:56:16,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:16,360 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:56:16,360 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:16,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53324 deadline: 1681315036359, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:16,361 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:56:16,361 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:56:16,361 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:56:16,361 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:16,361 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:56:16,362 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:56:16,362 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:56:16,362 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:16,362 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. started... 2023-04-12 15:56:16,362 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. as already flushing 2023-04-12 15:56:16,362 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:16,362 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:56:16,362 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:56:16,362 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:16,362 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:56:16,362 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:56:16,365 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:16,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:16,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:16,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:16,365 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:16,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:16,365 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:16,365 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:56:16,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:16,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:16,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:16,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:16,367 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:56:16,368 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a2beecd[Count = 0] remaining members to acquire global barrier 2023-04-12 15:56:16,368 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:56:16,368 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:16,369 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:16,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:16,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:16,370 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:56:16,370 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:56:16,370 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:56:16,370 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:16,370 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:56:16,370 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/C/8ef1107d19c94ac391510d308aa5589e as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/8ef1107d19c94ac391510d308aa5589e 2023-04-12 15:56:16,373 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:16,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:16,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:16,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:16,373 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:56:16,373 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:16,373 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:56:16,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:16,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:16,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:16,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:16,375 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4870003209f366bc50ef2084e6e2cf03/C of 4870003209f366bc50ef2084e6e2cf03 into 8ef1107d19c94ac391510d308aa5589e(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:56:16,376 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4870003209f366bc50ef2084e6e2cf03: 2023-04-12 15:56:16,376 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03., storeName=4870003209f366bc50ef2084e6e2cf03/C, priority=13, startTime=1681314976231; duration=0sec 2023-04-12 15:56:16,376 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:56:16,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:16,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:16,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:16,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:56:16,377 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:56:16,377 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:56:16,378 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:56:16,378 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:56:16,378 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:56:16,378 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:56:16,379 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:16,379 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:16,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:16,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:16,379 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:16,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:16,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:16,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:16,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:16,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:16,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:16,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:16,381 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:16,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:16,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:16,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:16,382 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:16,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:16,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:16,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:16,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:16,385 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:16,385 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:16,385 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:16,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:16,385 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:16,385 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:56:16,385 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:56:16,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:16,385 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:16,385 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:16,385 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:16,385 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:16,386 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:16,386 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:16,386 DEBUG [Thread-4110] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:56:16,386 DEBUG [Thread-4110] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:56:16,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:16,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:16,390 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=392 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/B/b422434fd7204aaa959242621aed1a9c 2023-04-12 15:56:16,402 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=392 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/C/66fcdac5789c40709a9317d4e01d4c3b 2023-04-12 15:56:16,407 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/A/7a8f17bfe827445f96297fa297e562a8 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/7a8f17bfe827445f96297fa297e562a8 2023-04-12 15:56:16,411 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/7a8f17bfe827445f96297fa297e562a8, entries=200, sequenceid=392, filesize=32.3 K 2023-04-12 15:56:16,412 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/B/b422434fd7204aaa959242621aed1a9c as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/b422434fd7204aaa959242621aed1a9c 2023-04-12 15:56:16,415 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/b422434fd7204aaa959242621aed1a9c, entries=150, sequenceid=392, filesize=11.9 K 2023-04-12 15:56:16,416 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/C/66fcdac5789c40709a9317d4e01d4c3b as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/66fcdac5789c40709a9317d4e01d4c3b 2023-04-12 15:56:16,422 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/66fcdac5789c40709a9317d4e01d4c3b, entries=150, sequenceid=392, filesize=11.9 K 2023-04-12 15:56:16,423 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 4870003209f366bc50ef2084e6e2cf03 in 187ms, sequenceid=392, compaction requested=false 2023-04-12 15:56:16,423 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4870003209f366bc50ef2084e6e2cf03: 2023-04-12 15:56:16,486 DEBUG [Thread-4110] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:56:16,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:56:16,488 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:56:16,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:56:16,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:56:16,496 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:56:16,496 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:16,496 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:56:16,496 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:56:16,497 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:16,497 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:16,498 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:16,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:16,498 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:16,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:16,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:16,499 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:16,499 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:16,499 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:56:16,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:56:16,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:16,500 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:56:16,501 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:56:16,501 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:56:16,501 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:56:16,501 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:16,501 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:56:16,504 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:56:16,504 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:56:16,504 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:16,504 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. started... 2023-04-12 15:56:16,504 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4870003209f366bc50ef2084e6e2cf03 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-12 15:56:16,504 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=A 2023-04-12 15:56:16,504 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:16,505 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=B 2023-04-12 15:56:16,505 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:16,505 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=C 2023-04-12 15:56:16,505 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:16,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on 4870003209f366bc50ef2084e6e2cf03 2023-04-12 15:56:16,561 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. as already flushing 2023-04-12 15:56:16,586 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:16,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53392 deadline: 1681315036582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:16,587 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:16,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53360 deadline: 1681315036585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:16,590 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:16,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53324 deadline: 1681315036587, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:16,689 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:16,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53392 deadline: 1681315036688, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:16,690 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:16,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53360 deadline: 1681315036688, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:16,693 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:16,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53324 deadline: 1681315036691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:16,757 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/A/107c4678604a4727bc1760700b46ae38 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/107c4678604a4727bc1760700b46ae38 2023-04-12 15:56:16,762 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 4870003209f366bc50ef2084e6e2cf03/A of 4870003209f366bc50ef2084e6e2cf03 into 107c4678604a4727bc1760700b46ae38(size=26.2 K), total size for store is 58.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:56:16,762 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4870003209f366bc50ef2084e6e2cf03: 2023-04-12 15:56:16,762 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03., storeName=4870003209f366bc50ef2084e6e2cf03/A, priority=11, startTime=1681314976231; duration=0sec 2023-04-12 15:56:16,763 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:56:16,893 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:16,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53360 deadline: 1681315036891, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:16,893 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:16,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53392 deadline: 1681315036891, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:16,897 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:16,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53324 deadline: 1681315036894, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:16,936 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230412f9867d4be0804de7a0c5d07a6b34dbcc to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412f9867d4be0804de7a0c5d07a6b34dbcc 2023-04-12 15:56:16,942 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=406, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/A/c67a737510e5471db76bc689782246c5 2023-04-12 15:56:16,951 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=406 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/B/faadc296ed7b45e5ba228f8d8eb77252 2023-04-12 15:56:16,960 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=406 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/C/b200901720474088a08fef2b53adad2c 2023-04-12 15:56:16,965 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/A/c67a737510e5471db76bc689782246c5 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/c67a737510e5471db76bc689782246c5 2023-04-12 15:56:16,968 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/c67a737510e5471db76bc689782246c5, entries=150, sequenceid=406, filesize=25.4 K 2023-04-12 15:56:16,969 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/B/faadc296ed7b45e5ba228f8d8eb77252 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/faadc296ed7b45e5ba228f8d8eb77252 2023-04-12 15:56:16,973 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/faadc296ed7b45e5ba228f8d8eb77252, entries=150, sequenceid=406, filesize=11.9 K 2023-04-12 15:56:16,973 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/C/b200901720474088a08fef2b53adad2c as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/b200901720474088a08fef2b53adad2c 2023-04-12 15:56:16,977 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/b200901720474088a08fef2b53adad2c, entries=150, sequenceid=406, filesize=11.9 K 2023-04-12 15:56:16,978 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 4870003209f366bc50ef2084e6e2cf03 in 474ms, sequenceid=406, compaction requested=true 2023-04-12 15:56:16,978 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4870003209f366bc50ef2084e6e2cf03: 2023-04-12 15:56:16,978 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:16,978 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:56:16,978 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:56:16,978 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:16,978 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:56:16,978 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:56:16,981 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:16,981 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:16,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:16,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:16,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:16,981 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:16,981 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:56:16,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:16,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:16,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:16,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:16,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:16,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:56:16,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5cb13fa6[Count = 0] remaining members to acquire global barrier 2023-04-12 15:56:16,983 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:56:16,983 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:16,984 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:16,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:16,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:16,984 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:56:16,984 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:56:16,984 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:56:16,985 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:16,985 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:56:16,986 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:56:16,986 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:16,986 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:56:16,986 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:16,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:16,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:16,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:16,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:16,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:16,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:16,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:16,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:16,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:16,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:16,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:56:16,990 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:56:16,990 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:56:16,990 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:56:16,990 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:56:16,990 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:56:16,990 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:56:16,992 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:16,992 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:16,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:16,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:16,992 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:16,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:16,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:16,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:16,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:16,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:16,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:16,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:16,994 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:16,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:16,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:16,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:16,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:16,996 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:16,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:16,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:16,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:16,999 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:16,999 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:16,999 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:16,999 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:17,000 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:17,000 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:56:16,999 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:17,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:17,000 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:17,000 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:56:17,000 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:56:17,000 DEBUG [Thread-4110] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:56:17,000 DEBUG [Thread-4110] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:56:17,000 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:17,000 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:17,000 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:17,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:17,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:17,100 DEBUG [Thread-4110] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:56:17,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:56:17,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:56:17,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:56:17,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:56:17,104 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:56:17,104 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:17,104 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:56:17,104 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:56:17,105 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:17,105 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:17,106 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,106 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:17,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:17,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:17,106 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,106 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:56:17,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:17,107 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:17,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:56:17,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:17,107 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:56:17,107 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:56:17,108 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:56:17,108 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:17,108 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:56:17,108 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:56:17,108 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:56:17,108 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:17,108 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. started... 2023-04-12 15:56:17,108 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4870003209f366bc50ef2084e6e2cf03 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-12 15:56:17,108 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=A 2023-04-12 15:56:17,109 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:17,109 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=B 2023-04-12 15:56:17,109 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:17,109 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=C 2023-04-12 15:56:17,109 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:17,150 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230412d20195d3990a450586e36c41152da10a to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412d20195d3990a450586e36c41152da10a 2023-04-12 15:56:17,162 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=427, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/A/ea344a8b54504ac6bef67542c932e792 2023-04-12 15:56:17,195 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=427 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/B/5b8d99a4d869480aad17d6a782c8a504 2023-04-12 15:56:17,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on 4870003209f366bc50ef2084e6e2cf03 2023-04-12 15:56:17,197 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. as already flushing 2023-04-12 15:56:17,208 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:17,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53324 deadline: 1681315037205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,209 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:17,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53392 deadline: 1681315037206, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,211 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:17,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53360 deadline: 1681315037209, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,213 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=427 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/C/1453d62d6ea34aef800f08e3a61292a9 2023-04-12 15:56:17,218 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/A/ea344a8b54504ac6bef67542c932e792 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/ea344a8b54504ac6bef67542c932e792 2023-04-12 15:56:17,221 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/ea344a8b54504ac6bef67542c932e792, entries=150, sequenceid=427, filesize=25.4 K 2023-04-12 15:56:17,222 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/B/5b8d99a4d869480aad17d6a782c8a504 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/5b8d99a4d869480aad17d6a782c8a504 2023-04-12 15:56:17,225 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/5b8d99a4d869480aad17d6a782c8a504, entries=150, sequenceid=427, filesize=11.9 K 2023-04-12 15:56:17,225 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/C/1453d62d6ea34aef800f08e3a61292a9 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/1453d62d6ea34aef800f08e3a61292a9 2023-04-12 15:56:17,231 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/1453d62d6ea34aef800f08e3a61292a9, entries=150, sequenceid=427, filesize=11.9 K 2023-04-12 15:56:17,232 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 4870003209f366bc50ef2084e6e2cf03 in 124ms, sequenceid=427, compaction requested=true 2023-04-12 15:56:17,232 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4870003209f366bc50ef2084e6e2cf03: 2023-04-12 15:56:17,232 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:17,232 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:56:17,232 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:56:17,232 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,232 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:56:17,232 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:56:17,234 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:17,234 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:17,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:17,234 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:17,234 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:56:17,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:17,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:17,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:17,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:17,236 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:56:17,236 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@453dc18a[Count = 0] remaining members to acquire global barrier 2023-04-12 15:56:17,236 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:56:17,236 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:17,238 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:17,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:17,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:17,238 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:56:17,238 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:56:17,238 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:56:17,238 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,238 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:56:17,240 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:17,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:17,240 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:56:17,241 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:17,241 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:56:17,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:17,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:17,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:17,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:17,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:17,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:56:17,243 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:56:17,243 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:56:17,243 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:56:17,243 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:56:17,243 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:56:17,243 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:56:17,245 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:17,245 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:17,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:17,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:17,245 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:17,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:17,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:17,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:17,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:17,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:17,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:17,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:17,246 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:17,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:17,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:17,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,248 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:17,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:17,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,251 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:17,251 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:17,251 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:56:17,251 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:56:17,251 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:17,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:17,251 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:17,251 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:17,251 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:17,251 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,251 DEBUG [Thread-4110] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:56:17,251 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:17,251 DEBUG [Thread-4110] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:56:17,251 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:17,251 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:17,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:17,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:17,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on 4870003209f366bc50ef2084e6e2cf03 2023-04-12 15:56:17,311 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4870003209f366bc50ef2084e6e2cf03 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-12 15:56:17,311 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=A 2023-04-12 15:56:17,312 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:17,312 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=B 2023-04-12 15:56:17,312 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:17,312 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=C 2023-04-12 15:56:17,312 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:17,324 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041241bb833ca39f49628a010066c7c0c880 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041241bb833ca39f49628a010066c7c0c880 2023-04-12 15:56:17,330 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=440, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/A/edab9cc0111f4945a386f1573d1efb40 2023-04-12 15:56:17,344 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:17,344 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=440 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/B/e33cb229559b46f78c7a1a99f691ab84 2023-04-12 15:56:17,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53360 deadline: 1681315037341, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,348 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:17,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53392 deadline: 1681315037343, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,348 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:17,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53324 deadline: 1681315037344, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,351 DEBUG [Thread-4110] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:56:17,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:56:17,354 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:56:17,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:56:17,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:56:17,356 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:56:17,356 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:17,356 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:56:17,356 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:56:17,356 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:17,356 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:17,358 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:17,358 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:17,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:17,359 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,359 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:56:17,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:17,359 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:17,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:56:17,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:17,360 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:56:17,360 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:56:17,360 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:56:17,360 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:56:17,361 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:17,361 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:56:17,361 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:56:17,361 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:56:17,361 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:17,361 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. started... 2023-04-12 15:56:17,361 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. as already flushing 2023-04-12 15:56:17,361 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:17,361 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:56:17,361 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:56:17,361 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,361 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:56:17,361 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:56:17,365 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:17,365 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:17,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:17,365 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:17,365 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:56:17,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:17,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:17,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:17,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:17,367 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:56:17,367 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1992affc[Count = 0] remaining members to acquire global barrier 2023-04-12 15:56:17,367 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:56:17,367 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:17,371 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:17,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:17,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:17,371 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:56:17,371 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,371 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:56:17,371 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:56:17,371 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:56:17,373 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:56:17,373 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:17,373 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:56:17,373 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:17,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:17,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:17,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:17,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:17,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:17,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:17,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:56:17,378 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:56:17,378 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:56:17,378 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:56:17,378 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:56:17,378 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:56:17,378 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:56:17,381 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:17,381 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:17,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:17,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:17,381 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:17,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:17,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:17,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:17,381 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:17,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:17,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:17,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:17,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:17,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:17,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:17,383 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:17,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:17,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,387 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,387 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:17,387 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:17,387 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:17,387 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:17,387 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:17,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:17,387 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:17,387 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:56:17,387 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:56:17,387 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:56:17,387 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:17,388 DEBUG [Thread-4110] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:56:17,388 DEBUG [Thread-4110] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:56:17,388 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:17,388 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:17,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:17,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:17,447 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:17,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53360 deadline: 1681315037445, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,452 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:17,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53392 deadline: 1681315037449, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,452 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:17,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53324 deadline: 1681315037450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,488 DEBUG [Thread-4110] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:56:17,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:56:17,490 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:56:17,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:56:17,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:56:17,491 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:56:17,491 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:17,491 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:56:17,491 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:56:17,492 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:17,492 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:17,493 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,494 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:17,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:17,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:17,494 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,494 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:56:17,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:17,495 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:17,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:56:17,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:17,495 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:56:17,495 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:56:17,495 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:56:17,495 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:56:17,495 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:17,495 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:56:17,496 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:56:17,496 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:56:17,496 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:17,496 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. started... 2023-04-12 15:56:17,496 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. as already flushing 2023-04-12 15:56:17,496 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:17,496 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:56:17,496 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:56:17,497 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,497 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:56:17,497 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:56:17,498 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,498 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:17,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:17,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:17,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:17,500 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:17,500 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:56:17,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:17,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:17,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:17,501 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:56:17,501 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c6a7785[Count = 0] remaining members to acquire global barrier 2023-04-12 15:56:17,501 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:56:17,501 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:17,502 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:17,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:17,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:17,503 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,503 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:56:17,503 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:56:17,503 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:56:17,503 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:56:17,506 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,506 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:56:17,506 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:17,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:17,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:17,506 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:56:17,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:17,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:17,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:17,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:17,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:17,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:56:17,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:56:17,509 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:56:17,509 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:56:17,509 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:56:17,509 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:56:17,509 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:56:17,510 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:17,510 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:17,510 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:17,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:17,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:17,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:17,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:17,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:17,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:17,511 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:17,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:17,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:17,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:17,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:17,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:17,513 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:17,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:17,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,516 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,516 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:17,516 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:17,516 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,516 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:17,516 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:17,516 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:17,516 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:17,517 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:17,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:17,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:17,517 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:56:17,517 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:56:17,517 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:17,517 DEBUG [Thread-4110] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:56:17,517 DEBUG [Thread-4110] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:56:17,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:17,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:17,617 DEBUG [Thread-4110] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:56:17,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:56:17,619 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:56:17,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:56:17,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:56:17,621 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:56:17,621 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:17,621 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:56:17,621 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:56:17,621 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:17,621 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:17,622 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,622 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:17,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:17,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:17,623 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,623 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:56:17,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:17,623 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:17,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:56:17,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:17,624 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:56:17,624 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:56:17,624 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:56:17,624 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:56:17,624 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:17,624 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:56:17,624 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:56:17,624 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:56:17,624 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:17,624 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. started... 2023-04-12 15:56:17,624 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. as already flushing 2023-04-12 15:56:17,624 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:17,624 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:56:17,625 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:56:17,625 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,625 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:56:17,625 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:56:17,627 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:17,627 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:17,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:17,628 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:17,628 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:56:17,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:17,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:17,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:17,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:17,629 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:56:17,629 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69f323a1[Count = 0] remaining members to acquire global barrier 2023-04-12 15:56:17,629 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:56:17,629 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:17,630 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:17,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:17,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:17,630 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:56:17,631 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:56:17,631 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:56:17,631 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,631 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:56:17,632 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:56:17,632 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:17,632 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,632 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:56:17,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:17,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:17,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:17,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:17,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:17,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:17,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:17,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:56:17,634 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:56:17,634 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:56:17,635 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:56:17,635 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:56:17,635 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:56:17,635 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:56:17,637 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:17,637 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:17,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:17,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:17,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:17,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:17,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:17,637 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:17,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:17,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:17,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:17,638 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:17,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:17,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:17,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:17,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,639 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:17,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:17,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,643 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,643 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:17,643 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:17,643 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:17,643 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:17,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:56:17,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:56:17,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:17,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:17,643 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:17,643 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,643 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:17,643 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:17,643 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:17,643 DEBUG [Thread-4110] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:56:17,643 DEBUG [Thread-4110] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:56:17,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:17,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:17,652 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:17,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53360 deadline: 1681315037650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,656 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:17,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53392 deadline: 1681315037653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,656 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:17,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53324 deadline: 1681315037654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,743 DEBUG [Thread-4110] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:56:17,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:56:17,745 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:56:17,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:56:17,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:56:17,747 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:56:17,747 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:17,747 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:56:17,747 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:56:17,747 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:17,747 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:17,748 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:17,748 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:17,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:17,749 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,749 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:56:17,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:17,749 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:17,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:56:17,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:17,749 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:56:17,750 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:56:17,750 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:56:17,750 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:56:17,750 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:17,750 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:56:17,750 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:56:17,750 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:56:17,750 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:17,750 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. started... 2023-04-12 15:56:17,750 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. as already flushing 2023-04-12 15:56:17,750 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:17,750 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:56:17,750 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:56:17,750 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,750 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:56:17,750 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:56:17,752 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,752 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:17,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:17,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:17,752 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:17,752 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:56:17,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:17,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:17,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:17,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:17,754 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:56:17,754 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42698e84[Count = 0] remaining members to acquire global barrier 2023-04-12 15:56:17,754 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:56:17,754 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:17,755 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:17,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:17,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:17,755 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:56:17,755 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:56:17,755 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:56:17,755 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,755 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:56:17,760 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:56:17,760 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:17,760 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,760 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:56:17,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:17,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:17,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:17,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:17,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:17,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:17,762 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=440 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/C/daed925ea716444a9fd864c060de4178 2023-04-12 15:56:17,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:17,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:56:17,763 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:56:17,763 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:56:17,763 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:56:17,763 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:56:17,763 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:56:17,764 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:56:17,765 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:17,765 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:17,765 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:17,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:17,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:17,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:17,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:17,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:17,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:17,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:17,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:17,766 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/A/edab9cc0111f4945a386f1573d1efb40 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/edab9cc0111f4945a386f1573d1efb40 2023-04-12 15:56:17,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:17,766 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:17,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:17,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:17,768 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:17,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:17,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,771 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:17,771 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,771 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:17,771 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:17,771 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:17,771 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:56:17,771 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:56:17,771 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:17,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:17,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:17,771 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,771 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:56:17,771 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:17,772 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:17,772 DEBUG [Thread-4110] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:56:17,772 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:17,772 DEBUG [Thread-4110] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:56:17,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:17,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:17,772 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/edab9cc0111f4945a386f1573d1efb40, entries=200, sequenceid=440, filesize=32.3 K 2023-04-12 15:56:17,773 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/B/e33cb229559b46f78c7a1a99f691ab84 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/e33cb229559b46f78c7a1a99f691ab84 2023-04-12 15:56:17,778 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/e33cb229559b46f78c7a1a99f691ab84, entries=150, sequenceid=440, filesize=11.9 K 2023-04-12 15:56:17,779 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/C/daed925ea716444a9fd864c060de4178 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/daed925ea716444a9fd864c060de4178 2023-04-12 15:56:17,785 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/daed925ea716444a9fd864c060de4178, entries=150, sequenceid=440, filesize=11.9 K 2023-04-12 15:56:17,785 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 4870003209f366bc50ef2084e6e2cf03 in 474ms, sequenceid=440, compaction requested=true 2023-04-12 15:56:17,785 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4870003209f366bc50ef2084e6e2cf03: 2023-04-12 15:56:17,786 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:56:17,786 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:56:17,786 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-12 15:56:17,786 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-12 15:56:17,786 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-12 15:56:17,787 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 145060 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-12 15:56:17,787 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1912): 4870003209f366bc50ef2084e6e2cf03/A is initiating minor compaction (all files) 2023-04-12 15:56:17,787 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4870003209f366bc50ef2084e6e2cf03/A in TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:17,787 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/107c4678604a4727bc1760700b46ae38, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/7a8f17bfe827445f96297fa297e562a8, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/c67a737510e5471db76bc689782246c5, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/ea344a8b54504ac6bef67542c932e792, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/edab9cc0111f4945a386f1573d1efb40] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp, totalSize=141.7 K 2023-04-12 15:56:17,788 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 107c4678604a4727bc1760700b46ae38, keycount=150, bloomtype=ROW, size=26.2 K, encoding=NONE, compression=NONE, seqNum=372, earliestPutTs=1681314975987 2023-04-12 15:56:17,788 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61714 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-12 15:56:17,788 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1912): 4870003209f366bc50ef2084e6e2cf03/B is initiating minor compaction (all files) 2023-04-12 15:56:17,788 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4870003209f366bc50ef2084e6e2cf03/B in TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:17,788 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/d13099c165374393873d1dc2ca8dda2e, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/b422434fd7204aaa959242621aed1a9c, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/faadc296ed7b45e5ba228f8d8eb77252, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/5b8d99a4d869480aad17d6a782c8a504, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/e33cb229559b46f78c7a1a99f691ab84] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp, totalSize=60.3 K 2023-04-12 15:56:17,789 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting d13099c165374393873d1dc2ca8dda2e, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=372, earliestPutTs=1681314975987 2023-04-12 15:56:17,789 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 7a8f17bfe827445f96297fa297e562a8, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=392, earliestPutTs=1681314976123 2023-04-12 15:56:17,789 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting b422434fd7204aaa959242621aed1a9c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=392, earliestPutTs=1681314976123 2023-04-12 15:56:17,789 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting c67a737510e5471db76bc689782246c5, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=406, earliestPutTs=1681314976248 2023-04-12 15:56:17,790 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting ea344a8b54504ac6bef67542c932e792, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=427, earliestPutTs=1681314976579 2023-04-12 15:56:17,790 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting faadc296ed7b45e5ba228f8d8eb77252, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=406, earliestPutTs=1681314976248 2023-04-12 15:56:17,790 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting edab9cc0111f4945a386f1573d1efb40, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=440, earliestPutTs=1681314977199 2023-04-12 15:56:17,790 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 5b8d99a4d869480aad17d6a782c8a504, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=427, earliestPutTs=1681314976579 2023-04-12 15:56:17,791 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting e33cb229559b46f78c7a1a99f691ab84, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=440, earliestPutTs=1681314977199 2023-04-12 15:56:17,801 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] throttle.PressureAwareThroughputController(145): 4870003209f366bc50ef2084e6e2cf03#B#compaction#636 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:56:17,804 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4870003209f366bc50ef2084e6e2cf03#A#compaction#635 average throughput is 3.29 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:56:17,872 DEBUG [Thread-4110] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:56:17,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:56:17,874 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:56:17,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:56:17,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:56:17,875 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:56:17,875 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:17,875 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:56:17,875 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:56:17,875 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:17,875 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:17,876 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,876 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:17,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:17,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:17,876 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,876 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:56:17,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:17,877 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:17,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:56:17,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:17,877 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:56:17,878 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:56:17,878 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:56:17,878 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:56:17,878 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:17,878 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:56:17,883 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:56:17,883 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:56:17,883 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:17,883 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. started... 2023-04-12 15:56:17,884 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4870003209f366bc50ef2084e6e2cf03 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-12 15:56:17,884 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=A 2023-04-12 15:56:17,884 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:17,884 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=B 2023-04-12 15:56:17,884 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:17,884 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=C 2023-04-12 15:56:17,884 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:17,905 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230412fdea57e19ae44beba16f68a41dc6d1b2 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412fdea57e19ae44beba16f68a41dc6d1b2 2023-04-12 15:56:17,915 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=459, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/A/77fe3a3d63c64de580f0330b14be5846 2023-04-12 15:56:17,924 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=459 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/B/85a304b6d02540129fced3ed0e5b71dc 2023-04-12 15:56:17,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on 4870003209f366bc50ef2084e6e2cf03 2023-04-12 15:56:17,955 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. as already flushing 2023-04-12 15:56:17,971 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:17,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53324 deadline: 1681315037968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,972 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:17,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53392 deadline: 1681315037969, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:17,972 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:17,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53360 deadline: 1681315037970, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:18,073 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:18,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53324 deadline: 1681315038072, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:18,075 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:18,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53392 deadline: 1681315038073, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:18,076 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:18,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53360 deadline: 1681315038074, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:18,222 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/B/3997ea0586fc4e699ffc15825b05c9fd as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/3997ea0586fc4e699ffc15825b05c9fd 2023-04-12 15:56:18,228 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 4870003209f366bc50ef2084e6e2cf03/B of 4870003209f366bc50ef2084e6e2cf03 into 3997ea0586fc4e699ffc15825b05c9fd(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:56:18,228 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4870003209f366bc50ef2084e6e2cf03: 2023-04-12 15:56:18,228 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03., storeName=4870003209f366bc50ef2084e6e2cf03/B, priority=11, startTime=1681314977786; duration=0sec 2023-04-12 15:56:18,228 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:56:18,228 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-12 15:56:18,235 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61680 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-12 15:56:18,236 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1912): 4870003209f366bc50ef2084e6e2cf03/C is initiating minor compaction (all files) 2023-04-12 15:56:18,236 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4870003209f366bc50ef2084e6e2cf03/C in TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:18,236 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/8ef1107d19c94ac391510d308aa5589e, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/66fcdac5789c40709a9317d4e01d4c3b, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/b200901720474088a08fef2b53adad2c, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/1453d62d6ea34aef800f08e3a61292a9, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/daed925ea716444a9fd864c060de4178] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp, totalSize=60.2 K 2023-04-12 15:56:18,236 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 8ef1107d19c94ac391510d308aa5589e, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=372, earliestPutTs=1681314975987 2023-04-12 15:56:18,236 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 66fcdac5789c40709a9317d4e01d4c3b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=392, earliestPutTs=1681314976123 2023-04-12 15:56:18,237 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting b200901720474088a08fef2b53adad2c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=406, earliestPutTs=1681314976248 2023-04-12 15:56:18,237 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 1453d62d6ea34aef800f08e3a61292a9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=427, earliestPutTs=1681314976579 2023-04-12 15:56:18,237 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting daed925ea716444a9fd864c060de4178, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=440, earliestPutTs=1681314977199 2023-04-12 15:56:18,248 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/A/37384b7eed344d19ae3c21ebeb4c7823 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/37384b7eed344d19ae3c21ebeb4c7823 2023-04-12 15:56:18,252 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 4870003209f366bc50ef2084e6e2cf03/A of 4870003209f366bc50ef2084e6e2cf03 into 37384b7eed344d19ae3c21ebeb4c7823(size=26.3 K), total size for store is 26.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:56:18,252 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4870003209f366bc50ef2084e6e2cf03: 2023-04-12 15:56:18,252 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03., storeName=4870003209f366bc50ef2084e6e2cf03/A, priority=11, startTime=1681314977786; duration=0sec 2023-04-12 15:56:18,252 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:56:18,253 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] throttle.PressureAwareThroughputController(145): 4870003209f366bc50ef2084e6e2cf03#C#compaction#640 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:56:18,276 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:18,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53324 deadline: 1681315038274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:18,280 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:18,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53392 deadline: 1681315038276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:18,280 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:18,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53360 deadline: 1681315038277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:18,338 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=459 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/C/7a8cc3dc8d32438bb6d343e749aa3144 2023-04-12 15:56:18,344 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/A/77fe3a3d63c64de580f0330b14be5846 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/77fe3a3d63c64de580f0330b14be5846 2023-04-12 15:56:18,347 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/77fe3a3d63c64de580f0330b14be5846, entries=150, sequenceid=459, filesize=25.4 K 2023-04-12 15:56:18,348 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/B/85a304b6d02540129fced3ed0e5b71dc as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/85a304b6d02540129fced3ed0e5b71dc 2023-04-12 15:56:18,352 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/85a304b6d02540129fced3ed0e5b71dc, entries=150, sequenceid=459, filesize=11.9 K 2023-04-12 15:56:18,352 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/C/7a8cc3dc8d32438bb6d343e749aa3144 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/7a8cc3dc8d32438bb6d343e749aa3144 2023-04-12 15:56:18,355 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/7a8cc3dc8d32438bb6d343e749aa3144, entries=150, sequenceid=459, filesize=11.9 K 2023-04-12 15:56:18,356 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 4870003209f366bc50ef2084e6e2cf03 in 472ms, sequenceid=459, compaction requested=false 2023-04-12 15:56:18,356 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4870003209f366bc50ef2084e6e2cf03: 2023-04-12 15:56:18,356 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:18,356 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:56:18,357 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:56:18,357 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:18,357 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:56:18,357 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:56:18,358 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:18,358 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:18,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:18,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:18,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:18,358 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:18,358 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:56:18,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:18,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:18,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:18,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:18,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:18,360 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:56:18,360 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e08347a[Count = 0] remaining members to acquire global barrier 2023-04-12 15:56:18,360 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:56:18,360 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:18,361 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:18,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:18,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:18,361 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:56:18,361 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:56:18,361 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:56:18,361 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:18,361 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:56:18,364 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:56:18,364 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:18,364 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:18,364 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:56:18,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:18,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:18,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:18,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:18,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:18,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:18,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:18,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:18,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:18,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:18,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:56:18,367 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:56:18,367 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:56:18,367 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:56:18,367 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:56:18,367 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:56:18,367 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:56:18,368 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:18,368 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:18,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:18,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:18,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:18,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:18,368 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:18,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:18,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:18,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:18,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:18,369 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:18,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:18,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:18,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:18,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:18,371 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:18,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:18,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:18,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:18,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:18,373 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:18,373 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:18,373 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:18,373 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:18,373 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:18,373 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:18,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:18,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:18,373 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:18,373 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:56:18,373 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:56:18,373 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:56:18,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:18,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:18,374 DEBUG [Thread-4110] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:56:18,374 DEBUG [Thread-4110] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:56:18,373 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:18,374 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:18,374 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:18,474 DEBUG [Thread-4110] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:56:18,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:56:18,476 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:56:18,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:56:18,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:56:18,478 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:56:18,478 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:18,478 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:56:18,478 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:56:18,478 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:18,478 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:18,479 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:18,479 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:18,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:18,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:18,479 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:18,480 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:56:18,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:18,480 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:18,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:56:18,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:18,480 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:56:18,480 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:56:18,481 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:56:18,481 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:56:18,481 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:18,481 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:56:18,481 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:56:18,481 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:56:18,481 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:18,481 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. started... 2023-04-12 15:56:18,481 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4870003209f366bc50ef2084e6e2cf03 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-12 15:56:18,482 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=A 2023-04-12 15:56:18,482 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:18,482 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=B 2023-04-12 15:56:18,482 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:18,482 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=C 2023-04-12 15:56:18,482 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:18,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on 4870003209f366bc50ef2084e6e2cf03 2023-04-12 15:56:18,580 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. as already flushing 2023-04-12 15:56:18,604 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:18,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53360 deadline: 1681315038602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:18,606 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:18,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53392 deadline: 1681315038604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:18,607 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:18,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53324 deadline: 1681315038605, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:18,664 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/C/1e75af02aefe4d5a9500e8be10838bfe as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/1e75af02aefe4d5a9500e8be10838bfe 2023-04-12 15:56:18,668 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 4870003209f366bc50ef2084e6e2cf03/C of 4870003209f366bc50ef2084e6e2cf03 into 1e75af02aefe4d5a9500e8be10838bfe(size=12.8 K), total size for store is 24.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:56:18,668 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4870003209f366bc50ef2084e6e2cf03: 2023-04-12 15:56:18,668 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03., storeName=4870003209f366bc50ef2084e6e2cf03/C, priority=11, startTime=1681314977786; duration=0sec 2023-04-12 15:56:18,668 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:56:18,706 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:18,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53360 deadline: 1681315038705, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:18,709 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:18,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53392 deadline: 1681315038707, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:18,709 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:18,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53324 deadline: 1681315038708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:18,893 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230412784e3113c20b4877b24d4f616dc0a7e1 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412784e3113c20b4877b24d4f616dc0a7e1 2023-04-12 15:56:18,899 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=474, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/A/6d076088d3e745a0aa8405a71460d178 2023-04-12 15:56:18,908 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=474 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/B/58a1617d011c4349931d177145ff64b1 2023-04-12 15:56:18,909 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:18,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53360 deadline: 1681315038908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:18,911 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:18,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53324 deadline: 1681315038911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:18,913 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:18,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53392 deadline: 1681315038911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:19,212 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:19,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53360 deadline: 1681315039211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:19,214 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:19,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53324 deadline: 1681315039213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:19,215 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:19,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53392 deadline: 1681315039214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:19,318 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=474 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/C/46593273575e4d59b310d87326d7b0dd 2023-04-12 15:56:19,321 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/A/6d076088d3e745a0aa8405a71460d178 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/6d076088d3e745a0aa8405a71460d178 2023-04-12 15:56:19,325 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/6d076088d3e745a0aa8405a71460d178, entries=150, sequenceid=474, filesize=25.4 K 2023-04-12 15:56:19,325 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/B/58a1617d011c4349931d177145ff64b1 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/58a1617d011c4349931d177145ff64b1 2023-04-12 15:56:19,329 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/58a1617d011c4349931d177145ff64b1, entries=150, sequenceid=474, filesize=11.9 K 2023-04-12 15:56:19,329 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/C/46593273575e4d59b310d87326d7b0dd as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/46593273575e4d59b310d87326d7b0dd 2023-04-12 15:56:19,333 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/46593273575e4d59b310d87326d7b0dd, entries=150, sequenceid=474, filesize=11.9 K 2023-04-12 15:56:19,333 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 4870003209f366bc50ef2084e6e2cf03 in 852ms, sequenceid=474, compaction requested=true 2023-04-12 15:56:19,333 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4870003209f366bc50ef2084e6e2cf03: 2023-04-12 15:56:19,333 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:19,333 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:56:19,334 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:56:19,334 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:19,334 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:56:19,334 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:56:19,336 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:19,336 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:19,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:19,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:19,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:19,336 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:19,337 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:56:19,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:19,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:19,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:19,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:19,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:19,338 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:56:19,338 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2131b461[Count = 0] remaining members to acquire global barrier 2023-04-12 15:56:19,338 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:56:19,338 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:19,339 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:19,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:19,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:19,339 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:56:19,339 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:56:19,339 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:56:19,340 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:19,340 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:56:19,341 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:56:19,341 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:19,341 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:19,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:19,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:19,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:19,341 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:56:19,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:19,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:19,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:19,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:19,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:19,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:19,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:19,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:56:19,353 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:56:19,353 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:56:19,353 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:56:19,353 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:56:19,353 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:56:19,353 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:56:19,354 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:19,355 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:19,355 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:19,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:19,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:19,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:19,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:19,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:19,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:19,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:19,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:19,355 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:19,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:19,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:19,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:19,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:19,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:19,356 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:19,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:19,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:19,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:19,360 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:19,360 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:19,360 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:19,360 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:19,360 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:56:19,360 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:56:19,360 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:19,360 DEBUG [Thread-4110] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:56:19,360 DEBUG [Thread-4110] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:56:19,360 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:56:19,360 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:19,360 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:19,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:19,361 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:19,361 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:19,361 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:19,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:19,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:19,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:19,460 DEBUG [Thread-4110] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:56:19,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:56:19,463 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:56:19,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:56:19,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:56:19,464 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:56:19,464 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:19,464 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:56:19,464 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:56:19,465 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:19,465 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:19,466 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:19,466 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:19,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:19,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:19,466 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:19,466 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:56:19,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:19,467 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:19,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:56:19,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:19,467 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:56:19,467 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:56:19,467 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:56:19,467 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:19,467 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:56:19,467 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:56:19,467 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:56:19,468 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:19,468 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. started... 2023-04-12 15:56:19,468 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4870003209f366bc50ef2084e6e2cf03 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-12 15:56:19,468 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=A 2023-04-12 15:56:19,468 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:19,468 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=B 2023-04-12 15:56:19,468 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:19,468 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=C 2023-04-12 15:56:19,468 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:19,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on 4870003209f366bc50ef2084e6e2cf03 2023-04-12 15:56:19,719 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. as already flushing 2023-04-12 15:56:19,730 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:19,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53392 deadline: 1681315039727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:19,730 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:19,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53360 deadline: 1681315039728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:19,732 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:19,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53324 deadline: 1681315039730, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:19,833 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:19,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53392 deadline: 1681315039831, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:19,833 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:19,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53360 deadline: 1681315039831, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:19,835 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:19,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53324 deadline: 1681315039833, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:19,881 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230412d18204bdaa4140dcb2691a8f1e13f5fe to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412d18204bdaa4140dcb2691a8f1e13f5fe 2023-04-12 15:56:19,892 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=494, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/A/ead3ca3fb60949cdbedf2441bc0acf93 2023-04-12 15:56:19,924 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=494 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/B/12631e3629864a439f00dd6e61aadb44 2023-04-12 15:56:19,938 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=494 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/C/0b9fdc14aec64f7e8ada8ad06eb674aa 2023-04-12 15:56:19,947 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/A/ead3ca3fb60949cdbedf2441bc0acf93 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/ead3ca3fb60949cdbedf2441bc0acf93 2023-04-12 15:56:19,952 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/ead3ca3fb60949cdbedf2441bc0acf93, entries=150, sequenceid=494, filesize=25.4 K 2023-04-12 15:56:19,953 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/B/12631e3629864a439f00dd6e61aadb44 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/12631e3629864a439f00dd6e61aadb44 2023-04-12 15:56:19,956 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/12631e3629864a439f00dd6e61aadb44, entries=150, sequenceid=494, filesize=11.9 K 2023-04-12 15:56:19,956 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/C/0b9fdc14aec64f7e8ada8ad06eb674aa as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/0b9fdc14aec64f7e8ada8ad06eb674aa 2023-04-12 15:56:19,960 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/0b9fdc14aec64f7e8ada8ad06eb674aa, entries=150, sequenceid=494, filesize=11.9 K 2023-04-12 15:56:19,961 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 4870003209f366bc50ef2084e6e2cf03 in 493ms, sequenceid=494, compaction requested=true 2023-04-12 15:56:19,961 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4870003209f366bc50ef2084e6e2cf03: 2023-04-12 15:56:19,961 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:19,961 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:56:19,961 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:56:19,961 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:19,961 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:56:19,961 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:56:19,963 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:19,963 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:19,963 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:56:19,964 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:19,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:19,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:19,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:19,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:19,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:19,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:19,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:19,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:19,966 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:56:19,966 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@512f65cd[Count = 0] remaining members to acquire global barrier 2023-04-12 15:56:19,966 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:56:19,966 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:19,967 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:19,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:19,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:19,967 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:56:19,967 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:56:19,967 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:19,967 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:56:19,967 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:56:19,969 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:19,969 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:56:19,969 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:19,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:19,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:19,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:19,969 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:56:19,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:19,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:19,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:19,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:19,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:19,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:19,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:19,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:56:19,972 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:56:19,972 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:56:19,972 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:56:19,972 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:56:19,973 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:56:19,973 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:56:19,974 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:19,974 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:19,974 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:19,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:19,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:19,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:19,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:19,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:19,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:19,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:19,975 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:19,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:19,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:19,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:19,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:19,976 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:19,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:19,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:19,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:19,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:19,981 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:19,981 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:19,981 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:19,981 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:19,981 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:19,981 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:19,981 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:56:19,981 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:56:19,981 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:19,981 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:56:19,981 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:19,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:19,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:19,981 DEBUG [Thread-4110] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:56:19,981 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:19,981 DEBUG [Thread-4110] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:56:19,981 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:19,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:19,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:20,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on 4870003209f366bc50ef2084e6e2cf03 2023-04-12 15:56:20,037 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4870003209f366bc50ef2084e6e2cf03 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-12 15:56:20,037 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=A 2023-04-12 15:56:20,037 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:20,037 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=B 2023-04-12 15:56:20,037 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:20,037 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=C 2023-04-12 15:56:20,037 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:20,049 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041291cf5c308ced41ecab7262cf301436e7 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041291cf5c308ced41ecab7262cf301436e7 2023-04-12 15:56:20,060 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:20,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53324 deadline: 1681315040058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:20,064 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:20,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53392 deadline: 1681315040060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:20,065 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:20,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53360 deadline: 1681315040060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:20,081 DEBUG [Thread-4110] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:56:20,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:56:20,083 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:56:20,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:56:20,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:56:20,084 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:56:20,084 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:20,084 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:56:20,084 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:56:20,085 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:20,085 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:20,087 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:20,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:20,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:20,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:20,087 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:20,088 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:20,088 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:20,088 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:56:20,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:56:20,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:20,088 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:56:20,088 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:56:20,088 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:56:20,088 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:56:20,088 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:20,088 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:56:20,089 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:56:20,089 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:56:20,089 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:20,089 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. started... 2023-04-12 15:56:20,089 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. as already flushing 2023-04-12 15:56:20,089 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:20,089 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:56:20,089 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:56:20,089 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:20,089 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:56:20,089 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:56:20,090 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:20,090 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:20,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:20,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:20,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:20,091 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:20,091 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:56:20,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:20,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:20,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:20,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:20,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:20,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:56:20,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5fbcdcc2[Count = 0] remaining members to acquire global barrier 2023-04-12 15:56:20,092 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:56:20,092 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:20,093 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:20,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:20,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:20,093 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:56:20,094 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:56:20,094 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:56:20,094 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:20,094 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:56:20,095 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:20,095 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:56:20,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:20,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:20,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:20,095 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:20,095 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:56:20,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:20,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:20,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:20,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:20,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:20,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:20,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:20,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:56:20,097 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:56:20,097 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:56:20,097 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:56:20,097 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:56:20,097 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:56:20,097 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:56:20,100 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:20,100 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:20,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:20,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:20,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:20,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:20,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:20,100 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:20,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:20,101 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:20,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:20,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:20,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:20,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:20,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:20,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:20,102 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:20,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:20,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:20,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:20,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:20,104 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:20,104 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:20,104 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:20,105 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:20,105 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:20,105 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:56:20,105 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:56:20,105 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:20,105 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:20,105 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:20,105 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:20,105 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:20,105 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:56:20,105 DEBUG [Thread-4110] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:56:20,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:20,105 DEBUG [Thread-4110] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:56:20,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:20,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:20,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:20,164 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:20,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53324 deadline: 1681315040161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:20,169 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:20,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53392 deadline: 1681315040166, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:20,169 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:20,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53360 deadline: 1681315040166, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:20,181 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:20,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53306 deadline: 1681315040178, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:20,183 DEBUG [Thread-4108] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8163 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03., hostname=jenkins-hbase4.apache.org,38905,1681314818739, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-12 15:56:20,205 DEBUG [Thread-4110] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:56:20,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:56:20,207 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:56:20,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:56:20,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:56:20,209 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:56:20,209 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:20,209 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:56:20,209 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:56:20,209 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:20,209 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:20,210 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:20,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53292 deadline: 1681315040209, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:20,210 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:20,210 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:20,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:20,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:20,212 DEBUG [Thread-4100] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8193 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03., hostname=jenkins-hbase4.apache.org,38905,1681314818739, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-12 15:56:20,212 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:20,212 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:56:20,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:20,212 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:20,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:56:20,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:20,213 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:56:20,213 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:56:20,213 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:56:20,213 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:56:20,213 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:20,213 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:56:20,213 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:56:20,213 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:56:20,213 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:20,213 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. started... 2023-04-12 15:56:20,213 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. as already flushing 2023-04-12 15:56:20,213 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:20,213 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:56:20,213 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:56:20,214 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:20,214 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:56:20,214 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:56:20,215 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:20,215 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:20,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:20,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:20,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:20,215 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:20,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:20,215 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:56:20,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:20,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:20,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:20,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:20,217 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:56:20,217 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@501888ab[Count = 0] remaining members to acquire global barrier 2023-04-12 15:56:20,217 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:56:20,217 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:20,219 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:20,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:20,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:20,219 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:56:20,219 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:56:20,219 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:20,219 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:56:20,219 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:56:20,220 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:20,220 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:56:20,220 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:20,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:20,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:20,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:20,220 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:56:20,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:20,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:20,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:20,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:20,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:20,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:20,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:20,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:56:20,223 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:56:20,223 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:56:20,223 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:56:20,223 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:56:20,223 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:56:20,223 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:56:20,224 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:20,224 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:20,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:20,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:20,224 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:20,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:20,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:20,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:20,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:20,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:20,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:20,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:20,225 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:20,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:20,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:20,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:20,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:20,227 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:20,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:20,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:20,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:20,228 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:20,228 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:20,228 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:20,229 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:56:20,229 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:56:20,229 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:20,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:20,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:20,229 DEBUG [Thread-4110] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:56:20,229 DEBUG [Thread-4110] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:56:20,229 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:20,229 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:20,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:20,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:20,229 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:20,229 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:20,229 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:20,229 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:20,329 DEBUG [Thread-4110] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:56:20,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:56:20,331 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:56:20,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:56:20,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:56:20,333 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:56:20,333 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:20,333 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:56:20,333 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:56:20,333 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:20,333 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:20,336 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:20,336 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:20,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:20,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:20,336 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:20,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:20,336 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:56:20,337 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:20,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:56:20,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:20,337 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:56:20,337 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:56:20,337 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:56:20,337 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:56:20,337 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:20,337 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:56:20,338 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:56:20,338 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:56:20,338 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:20,338 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. started... 2023-04-12 15:56:20,338 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. as already flushing 2023-04-12 15:56:20,338 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:20,338 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:56:20,338 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:56:20,338 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:20,338 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:56:20,338 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:56:20,339 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:20,339 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:20,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:20,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:20,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:20,340 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:20,340 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:56:20,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:20,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:20,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:20,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:20,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:20,341 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:56:20,341 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ca6bc75[Count = 0] remaining members to acquire global barrier 2023-04-12 15:56:20,341 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:56:20,341 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:20,342 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:20,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:20,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:20,343 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:56:20,343 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:56:20,343 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:56:20,343 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:20,343 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:56:20,344 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:56:20,344 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:20,344 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:56:20,344 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:20,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:20,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:20,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:20,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:20,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:20,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:20,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:20,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:20,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:20,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:20,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:56:20,346 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:56:20,346 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:56:20,347 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:56:20,347 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:56:20,347 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:56:20,347 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:56:20,349 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:20,349 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:20,349 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:20,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:20,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:20,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:20,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:20,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:20,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:20,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:20,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:20,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:20,350 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:20,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:20,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:20,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:20,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:20,351 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:20,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:20,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:20,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:20,356 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:20,356 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:56:20,356 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:56:20,356 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:20,356 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:20,356 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:20,356 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:20,356 DEBUG [Thread-4110] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:56:20,356 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:20,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:20,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:20,357 DEBUG [Thread-4110] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:56:20,357 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:20,357 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:20,357 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:20,357 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:20,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:20,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:20,366 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:20,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53324 deadline: 1681315040365, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:20,371 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:20,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53392 deadline: 1681315040370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:20,373 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:20,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53360 deadline: 1681315040371, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:20,457 DEBUG [Thread-4110] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:56:20,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:56:20,458 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=509, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/A/afb9c740adc7457083daeb07819b3a27 2023-04-12 15:56:20,459 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:56:20,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:56:20,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:56:20,460 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:56:20,460 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:20,460 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:56:20,460 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:56:20,461 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:20,461 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:20,462 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:20,462 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:20,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:20,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:20,463 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:20,463 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:56:20,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:20,463 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:20,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:56:20,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:20,464 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:56:20,464 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:56:20,464 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:56:20,464 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:56:20,464 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:20,464 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:56:20,465 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:56:20,465 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:56:20,465 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:20,465 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. started... 2023-04-12 15:56:20,466 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. as already flushing 2023-04-12 15:56:20,466 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:20,466 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:56:20,466 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:56:20,466 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:20,466 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:56:20,466 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:56:20,468 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:20,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:20,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:20,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:20,468 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:20,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:20,469 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:20,469 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:56:20,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:20,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:20,469 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=509 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/B/11c2c117cc2e481a8f06292d81e491e7 2023-04-12 15:56:20,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:20,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:20,470 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:56:20,470 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@568daf21[Count = 0] remaining members to acquire global barrier 2023-04-12 15:56:20,470 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:56:20,470 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:20,473 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:20,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:20,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:20,473 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:56:20,473 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:56:20,473 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:56:20,474 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:20,474 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:56:20,475 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:56:20,475 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:20,475 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:56:20,475 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:20,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:20,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:20,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:20,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:20,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:20,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:20,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:20,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:20,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:20,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:20,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:56:20,478 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:56:20,478 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:56:20,478 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:56:20,478 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:56:20,478 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:56:20,478 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:56:20,479 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:20,480 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:20,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:20,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:20,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:20,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:20,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:20,480 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:20,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:20,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:20,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:20,481 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:20,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:20,481 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=509 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/C/924a62afc9264e7eabb1b8d9ba76f108 2023-04-12 15:56:20,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:20,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:20,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:20,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:20,483 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:20,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:20,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:20,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:20,486 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:20,486 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:20,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:20,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:20,486 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:20,486 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/A/afb9c740adc7457083daeb07819b3a27 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/afb9c740adc7457083daeb07819b3a27 2023-04-12 15:56:20,486 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:56:20,486 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:56:20,486 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:20,486 DEBUG [Thread-4110] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:56:20,486 DEBUG [Thread-4110] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:56:20,486 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:20,486 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:20,486 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:20,486 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:20,486 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:20,486 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:20,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:20,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:20,487 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:56:20,490 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/afb9c740adc7457083daeb07819b3a27, entries=250, sequenceid=509, filesize=39.2 K 2023-04-12 15:56:20,490 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/B/11c2c117cc2e481a8f06292d81e491e7 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/11c2c117cc2e481a8f06292d81e491e7 2023-04-12 15:56:20,494 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/11c2c117cc2e481a8f06292d81e491e7, entries=150, sequenceid=509, filesize=11.9 K 2023-04-12 15:56:20,495 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/C/924a62afc9264e7eabb1b8d9ba76f108 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/924a62afc9264e7eabb1b8d9ba76f108 2023-04-12 15:56:20,498 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/924a62afc9264e7eabb1b8d9ba76f108, entries=150, sequenceid=509, filesize=11.9 K 2023-04-12 15:56:20,499 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=100.63 KB/103050 for 4870003209f366bc50ef2084e6e2cf03 in 462ms, sequenceid=509, compaction requested=true 2023-04-12 15:56:20,499 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4870003209f366bc50ef2084e6e2cf03: 2023-04-12 15:56:20,499 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:56:20,499 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-12 15:56:20,499 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-12 15:56:20,499 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:56:20,499 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:56:20,501 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61884 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-12 15:56:20,501 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1912): 4870003209f366bc50ef2084e6e2cf03/B is initiating minor compaction (all files) 2023-04-12 15:56:20,501 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4870003209f366bc50ef2084e6e2cf03/B in TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:20,501 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 145234 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-12 15:56:20,501 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/3997ea0586fc4e699ffc15825b05c9fd, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/85a304b6d02540129fced3ed0e5b71dc, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/58a1617d011c4349931d177145ff64b1, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/12631e3629864a439f00dd6e61aadb44, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/11c2c117cc2e481a8f06292d81e491e7] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp, totalSize=60.4 K 2023-04-12 15:56:20,501 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1912): 4870003209f366bc50ef2084e6e2cf03/A is initiating minor compaction (all files) 2023-04-12 15:56:20,502 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4870003209f366bc50ef2084e6e2cf03/A in TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:20,502 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/37384b7eed344d19ae3c21ebeb4c7823, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/77fe3a3d63c64de580f0330b14be5846, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/6d076088d3e745a0aa8405a71460d178, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/ead3ca3fb60949cdbedf2441bc0acf93, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/afb9c740adc7457083daeb07819b3a27] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp, totalSize=141.8 K 2023-04-12 15:56:20,502 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 3997ea0586fc4e699ffc15825b05c9fd, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=440, earliestPutTs=1681314977199 2023-04-12 15:56:20,502 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 37384b7eed344d19ae3c21ebeb4c7823, keycount=150, bloomtype=ROW, size=26.3 K, encoding=NONE, compression=NONE, seqNum=440, earliestPutTs=1681314977199 2023-04-12 15:56:20,502 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 85a304b6d02540129fced3ed0e5b71dc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=459, earliestPutTs=1681314977339 2023-04-12 15:56:20,502 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 77fe3a3d63c64de580f0330b14be5846, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=459, earliestPutTs=1681314977339 2023-04-12 15:56:20,502 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 58a1617d011c4349931d177145ff64b1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=474, earliestPutTs=1681314977966 2023-04-12 15:56:20,503 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 6d076088d3e745a0aa8405a71460d178, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=474, earliestPutTs=1681314977966 2023-04-12 15:56:20,503 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 12631e3629864a439f00dd6e61aadb44, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=494, earliestPutTs=1681314978595 2023-04-12 15:56:20,503 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting ead3ca3fb60949cdbedf2441bc0acf93, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=494, earliestPutTs=1681314978595 2023-04-12 15:56:20,503 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 11c2c117cc2e481a8f06292d81e491e7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=509, earliestPutTs=1681314979727 2023-04-12 15:56:20,503 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting afb9c740adc7457083daeb07819b3a27, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=509, earliestPutTs=1681314979726 2023-04-12 15:56:20,511 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] throttle.PressureAwareThroughputController(145): 4870003209f366bc50ef2084e6e2cf03#B#compaction#650 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:56:20,517 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4870003209f366bc50ef2084e6e2cf03#A#compaction#651 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:56:20,586 DEBUG [Thread-4110] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:56:20,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:56:20,588 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:56:20,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:56:20,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:56:20,589 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:56:20,589 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:20,589 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:56:20,589 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:56:20,590 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:20,590 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:20,592 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:20,592 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:20,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:20,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:20,593 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:20,593 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:56:20,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:20,593 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:20,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:56:20,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:20,594 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:56:20,594 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:56:20,594 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:56:20,594 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:56:20,594 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:20,594 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:56:20,594 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:56:20,594 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:56:20,594 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:20,594 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. started... 2023-04-12 15:56:20,594 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4870003209f366bc50ef2084e6e2cf03 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-12 15:56:20,595 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=A 2023-04-12 15:56:20,595 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:20,595 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=B 2023-04-12 15:56:20,595 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:20,595 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=C 2023-04-12 15:56:20,595 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:20,610 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041200576d01df0e48978bb6fa7e1302d2ce to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041200576d01df0e48978bb6fa7e1302d2ce 2023-04-12 15:56:20,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on 4870003209f366bc50ef2084e6e2cf03 2023-04-12 15:56:20,669 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. as already flushing 2023-04-12 15:56:20,691 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:20,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53392 deadline: 1681315040687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:20,691 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:20,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53324 deadline: 1681315040688, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:20,694 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:20,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53360 deadline: 1681315040690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:20,795 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:20,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53392 deadline: 1681315040792, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:20,795 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:20,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53324 deadline: 1681315040792, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:20,795 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:20,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53360 deadline: 1681315040794, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:20,919 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/B/474092d4fe3142ec986b7887effde366 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/474092d4fe3142ec986b7887effde366 2023-04-12 15:56:20,929 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 4870003209f366bc50ef2084e6e2cf03/B of 4870003209f366bc50ef2084e6e2cf03 into 474092d4fe3142ec986b7887effde366(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:56:20,929 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4870003209f366bc50ef2084e6e2cf03: 2023-04-12 15:56:20,929 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03., storeName=4870003209f366bc50ef2084e6e2cf03/B, priority=11, startTime=1681314980499; duration=0sec 2023-04-12 15:56:20,929 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:56:20,929 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-12 15:56:20,931 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61850 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-12 15:56:20,931 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1912): 4870003209f366bc50ef2084e6e2cf03/C is initiating minor compaction (all files) 2023-04-12 15:56:20,931 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4870003209f366bc50ef2084e6e2cf03/C in TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:20,932 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/1e75af02aefe4d5a9500e8be10838bfe, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/7a8cc3dc8d32438bb6d343e749aa3144, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/46593273575e4d59b310d87326d7b0dd, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/0b9fdc14aec64f7e8ada8ad06eb674aa, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/924a62afc9264e7eabb1b8d9ba76f108] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp, totalSize=60.4 K 2023-04-12 15:56:20,932 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 1e75af02aefe4d5a9500e8be10838bfe, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=440, earliestPutTs=1681314977199 2023-04-12 15:56:20,933 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 7a8cc3dc8d32438bb6d343e749aa3144, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=459, earliestPutTs=1681314977339 2023-04-12 15:56:20,933 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 46593273575e4d59b310d87326d7b0dd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=474, earliestPutTs=1681314977966 2023-04-12 15:56:20,934 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 0b9fdc14aec64f7e8ada8ad06eb674aa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=494, earliestPutTs=1681314978595 2023-04-12 15:56:20,934 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 924a62afc9264e7eabb1b8d9ba76f108, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=509, earliestPutTs=1681314979727 2023-04-12 15:56:20,942 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] throttle.PressureAwareThroughputController(145): 4870003209f366bc50ef2084e6e2cf03#C#compaction#653 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:56:20,947 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/A/2d948b679bd44a318a5eb0b0558b9384 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/2d948b679bd44a318a5eb0b0558b9384 2023-04-12 15:56:20,951 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 4870003209f366bc50ef2084e6e2cf03/A of 4870003209f366bc50ef2084e6e2cf03 into 2d948b679bd44a318a5eb0b0558b9384(size=26.5 K), total size for store is 26.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:56:20,951 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4870003209f366bc50ef2084e6e2cf03: 2023-04-12 15:56:20,951 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03., storeName=4870003209f366bc50ef2084e6e2cf03/A, priority=11, startTime=1681314980499; duration=0sec 2023-04-12 15:56:20,951 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:56:20,953 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/C/5218664995954f6a97811f3158dc1bb7 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/5218664995954f6a97811f3158dc1bb7 2023-04-12 15:56:20,956 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 4870003209f366bc50ef2084e6e2cf03/C of 4870003209f366bc50ef2084e6e2cf03 into 5218664995954f6a97811f3158dc1bb7(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:56:20,956 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4870003209f366bc50ef2084e6e2cf03: 2023-04-12 15:56:20,956 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03., storeName=4870003209f366bc50ef2084e6e2cf03/C, priority=11, startTime=1681314980499; duration=0sec 2023-04-12 15:56:20,956 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:56:20,997 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:20,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53392 deadline: 1681315040996, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:20,997 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:20,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53324 deadline: 1681315040996, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:20,998 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:20,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53360 deadline: 1681315040997, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:21,015 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=527, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/A/77e3e05f931241509e187715a572dc97 2023-04-12 15:56:21,042 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=527 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/B/6e5f54175c004e44bb5e5ee928cdf2c4 2023-04-12 15:56:21,301 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:21,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53392 deadline: 1681315041299, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:21,301 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:21,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53324 deadline: 1681315041300, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:21,302 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:21,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53360 deadline: 1681315041300, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:21,451 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=527 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/C/df3761fa0d3c4398bc41dd3ff9626f86 2023-04-12 15:56:21,455 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/A/77e3e05f931241509e187715a572dc97 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/77e3e05f931241509e187715a572dc97 2023-04-12 15:56:21,458 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/77e3e05f931241509e187715a572dc97, entries=150, sequenceid=527, filesize=25.4 K 2023-04-12 15:56:21,459 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/B/6e5f54175c004e44bb5e5ee928cdf2c4 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/6e5f54175c004e44bb5e5ee928cdf2c4 2023-04-12 15:56:21,462 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/6e5f54175c004e44bb5e5ee928cdf2c4, entries=150, sequenceid=527, filesize=11.9 K 2023-04-12 15:56:21,463 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/C/df3761fa0d3c4398bc41dd3ff9626f86 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/df3761fa0d3c4398bc41dd3ff9626f86 2023-04-12 15:56:21,468 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/df3761fa0d3c4398bc41dd3ff9626f86, entries=150, sequenceid=527, filesize=11.9 K 2023-04-12 15:56:21,468 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 4870003209f366bc50ef2084e6e2cf03 in 874ms, sequenceid=527, compaction requested=false 2023-04-12 15:56:21,468 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4870003209f366bc50ef2084e6e2cf03: 2023-04-12 15:56:21,468 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:21,468 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:56:21,468 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:56:21,468 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:21,472 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:56:21,472 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:56:21,473 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:21,473 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:21,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:21,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:21,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:21,474 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:21,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:21,474 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:56:21,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:21,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:21,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:21,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:21,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:56:21,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6fee23fe[Count = 0] remaining members to acquire global barrier 2023-04-12 15:56:21,475 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:56:21,475 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:21,476 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:21,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:21,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:21,476 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:56:21,477 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:56:21,477 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:56:21,477 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:21,477 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:56:21,480 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:56:21,480 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:21,480 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:21,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:21,480 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:56:21,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:21,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:21,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:21,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:21,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:21,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:21,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:21,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:21,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:21,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:56:21,483 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:56:21,483 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:56:21,483 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:56:21,483 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:56:21,483 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:56:21,483 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:56:21,484 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:21,484 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:21,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:21,484 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:21,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:21,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:21,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:21,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:21,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:21,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:21,485 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:21,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:21,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:21,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:21,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:21,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:21,486 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:21,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:21,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:21,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:21,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:21,488 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:21,488 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:21,488 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:21,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:21,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:21,489 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:21,489 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:21,489 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:21,489 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:21,489 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:21,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:21,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:21,489 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:21,489 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:21,489 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:56:21,489 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:56:21,489 DEBUG [Thread-4110] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:56:21,489 DEBUG [Thread-4110] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:56:21,590 DEBUG [Thread-4110] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:56:21,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:56:21,592 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:56:21,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:56:21,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:56:21,593 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:56:21,593 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:21,593 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:56:21,593 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:56:21,594 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:21,594 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:21,595 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:21,595 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:21,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:21,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:21,595 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:21,595 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:56:21,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:21,596 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:21,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:56:21,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:21,596 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:56:21,597 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:56:21,597 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:56:21,597 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:21,597 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:56:21,597 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:56:21,597 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:56:21,597 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:21,597 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. started... 2023-04-12 15:56:21,597 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4870003209f366bc50ef2084e6e2cf03 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-12 15:56:21,598 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=A 2023-04-12 15:56:21,598 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:21,598 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=B 2023-04-12 15:56:21,598 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:21,598 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=C 2023-04-12 15:56:21,598 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:21,616 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230412304c09afebe945d0aa5fefd168b5ccc7 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412304c09afebe945d0aa5fefd168b5ccc7 2023-04-12 15:56:21,624 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=544, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/A/36d9863e74c44fd9858afafe8a72e980 2023-04-12 15:56:21,642 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=544 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/B/e67caf640b0545e1a783f81b3655d89a 2023-04-12 15:56:21,651 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=544 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/C/663db4c933884290a21af172f7a34a3a 2023-04-12 15:56:21,656 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/A/36d9863e74c44fd9858afafe8a72e980 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/36d9863e74c44fd9858afafe8a72e980 2023-04-12 15:56:21,663 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/36d9863e74c44fd9858afafe8a72e980, entries=150, sequenceid=544, filesize=25.4 K 2023-04-12 15:56:21,664 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/B/e67caf640b0545e1a783f81b3655d89a as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/e67caf640b0545e1a783f81b3655d89a 2023-04-12 15:56:21,667 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/e67caf640b0545e1a783f81b3655d89a, entries=150, sequenceid=544, filesize=11.9 K 2023-04-12 15:56:21,668 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/C/663db4c933884290a21af172f7a34a3a as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/663db4c933884290a21af172f7a34a3a 2023-04-12 15:56:21,689 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/663db4c933884290a21af172f7a34a3a, entries=150, sequenceid=544, filesize=11.9 K 2023-04-12 15:56:21,689 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=0 B/0 for 4870003209f366bc50ef2084e6e2cf03 in 92ms, sequenceid=544, compaction requested=true 2023-04-12 15:56:21,690 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4870003209f366bc50ef2084e6e2cf03: 2023-04-12 15:56:21,690 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:21,690 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:56:21,690 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:56:21,690 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:21,690 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:56:21,690 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:56:21,692 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:21,692 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:21,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:21,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:21,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:21,692 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:21,693 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:56:21,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:21,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:21,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:21,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:21,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:21,695 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:56:21,695 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ba7b378[Count = 0] remaining members to acquire global barrier 2023-04-12 15:56:21,695 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:56:21,695 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:21,696 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:21,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:21,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:21,696 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:56:21,696 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:56:21,696 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:56:21,696 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:21,696 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:56:21,698 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:21,698 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:56:21,698 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:21,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:21,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:21,698 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:56:21,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:21,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:21,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:21,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:21,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:21,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:21,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:21,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:21,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:56:21,701 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:56:21,701 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:56:21,701 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:56:21,701 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:56:21,701 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:56:21,701 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:56:21,703 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:21,703 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:21,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:21,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:21,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:21,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:21,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:21,703 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:21,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:21,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:21,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:21,704 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:21,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:21,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:21,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:21,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:21,705 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:21,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:21,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:21,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:21,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:21,708 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:21,709 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:21,709 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:21,709 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:21,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:21,709 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:56:21,709 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:56:21,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:21,709 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:21,709 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:21,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:21,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:21,709 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:21,709 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:21,709 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:21,709 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:21,709 DEBUG [Thread-4110] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:56:21,709 DEBUG [Thread-4110] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:56:21,810 DEBUG [Thread-4110] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:56:21,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:56:21,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:56:21,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:56:21,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:56:21,813 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:56:21,813 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:21,813 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:56:21,813 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:56:21,813 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:21,814 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:21,818 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:21,818 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:21,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:21,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:21,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:21,818 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:21,818 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:56:21,819 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:21,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:56:21,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:21,819 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:56:21,828 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:56:21,828 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:56:21,828 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:56:21,828 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:21,828 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:56:21,828 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:56:21,828 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:56:21,828 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:21,828 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. started... 2023-04-12 15:56:21,829 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4870003209f366bc50ef2084e6e2cf03 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-12 15:56:21,829 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=A 2023-04-12 15:56:21,829 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:21,829 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=B 2023-04-12 15:56:21,829 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:21,829 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=C 2023-04-12 15:56:21,829 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:21,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on 4870003209f366bc50ef2084e6e2cf03 2023-04-12 15:56:21,848 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. as already flushing 2023-04-12 15:56:21,856 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304121af173b3e174448fba7b6a1ec217acdc to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304121af173b3e174448fba7b6a1ec217acdc 2023-04-12 15:56:21,872 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681314981864, max timestamp=1681314981870Num uniques 859; 2023-04-12 15:56:21,872 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681314981864, max timestamp=1681314981870Num uniques 859; 2023-04-12 15:56:21,873 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-12 15:56:21,872 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681314981864, max timestamp=1681314981870Num uniques 859; 2023-04-12 15:56:21,873 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-12 15:56:21,872 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-12 15:56:21,900 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681314981883, max timestamp=1681314981898Num uniques 1118; 2023-04-12 15:56:21,901 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681314981883, max timestamp=1681314981898Num uniques 1118; 2023-04-12 15:56:21,901 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=9978, suffix heap size=144240,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=218 2023-04-12 15:56:21,901 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681314981883, max timestamp=1681314981898Num uniques 1118; 2023-04-12 15:56:21,901 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=9978, suffix heap size=144240,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=218 2023-04-12 15:56:21,901 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=9978, suffix heap size=144240,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=218 2023-04-12 15:56:21,924 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1681314981916, max timestamp=1681314981921Num uniques 1077; 2023-04-12 15:56:21,924 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-04-12 15:56:21,924 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1681314981916, max timestamp=1681314981921Num uniques 1077; 2023-04-12 15:56:21,924 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1681314981916, max timestamp=1681314981921Num uniques 1077; 2023-04-12 15:56:21,924 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-04-12 15:56:21,924 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-04-12 15:56:21,948 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1681314981935, max timestamp=1681314981945Num uniques 1086; 2023-04-12 15:56:21,948 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1681314981935, max timestamp=1681314981945Num uniques 1086; 2023-04-12 15:56:21,948 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1681314981940, max timestamp=1681314981945Num uniques 1086; 2023-04-12 15:56:21,948 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=13096, suffix heap size=140912,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=286 2023-04-12 15:56:21,948 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=13096, suffix heap size=140912,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=286 2023-04-12 15:56:21,948 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-04-12 15:56:21,973 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1681314981961, max timestamp=1681314981970Num uniques 1145; 2023-04-12 15:56:21,973 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1681314981961, max timestamp=1681314981970Num uniques 1145; 2023-04-12 15:56:21,973 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1681314981961, max timestamp=1681314981970Num uniques 1095; 2023-04-12 15:56:21,973 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=13510, suffix heap size=141848,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=295 2023-04-12 15:56:21,973 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=13510, suffix heap size=147048,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=295 2023-04-12 15:56:21,973 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=13510, suffix heap size=147048,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=295 2023-04-12 15:56:22,001 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1681314981991, max timestamp=1681314981999Num uniques 1154; 2023-04-12 15:56:22,001 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=11630, suffix heap size=147984,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=254 2023-04-12 15:56:22,002 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1681314981991, max timestamp=1681314981999Num uniques 1154; 2023-04-12 15:56:22,002 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=11630, suffix heap size=147984,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=254 2023-04-12 15:56:22,004 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1681314981992, max timestamp=1681314981999Num uniques 1154; 2023-04-12 15:56:22,004 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=7050, suffix heap size=147984,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=154 2023-04-12 15:56:22,031 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1681314982013, max timestamp=1681314982027Num uniques 1113; 2023-04-12 15:56:22,031 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=9748, suffix heap size=143720,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=213 2023-04-12 15:56:22,031 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1681314982013, max timestamp=1681314982027Num uniques 1113; 2023-04-12 15:56:22,031 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1681314982013, max timestamp=1681314982027Num uniques 1013; 2023-04-12 15:56:22,031 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=9748, suffix heap size=143720,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=213 2023-04-12 15:56:22,031 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=9748, suffix heap size=133320,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=213 2023-04-12 15:56:22,049 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1681314982040, max timestamp=1681314982048Num uniques 1072; 2023-04-12 15:56:22,049 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=10162, suffix heap size=139456,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=222 2023-04-12 15:56:22,050 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1681314982040, max timestamp=1681314982048Num uniques 1072; 2023-04-12 15:56:22,050 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=10162, suffix heap size=139456,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=222 2023-04-12 15:56:22,050 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1681314982040, max timestamp=1681314982048Num uniques 1072; 2023-04-12 15:56:22,050 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=10162, suffix heap size=139456,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=222 2023-04-12 15:56:22,072 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1681314982066, max timestamp=1681314982070Num uniques 1081; 2023-04-12 15:56:22,072 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=10576, suffix heap size=140392,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=231 2023-04-12 15:56:22,072 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1681314982066, max timestamp=1681314982070Num uniques 1081; 2023-04-12 15:56:22,073 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=10576, suffix heap size=140392,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=231 2023-04-12 15:56:22,074 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1681314982066, max timestamp=1681314982070Num uniques 1081; 2023-04-12 15:56:22,074 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=10576, suffix heap size=140392,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=231 2023-04-12 15:56:22,098 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1681314982089, max timestamp=1681314982096Num uniques 1090; 2023-04-12 15:56:22,098 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=10990, suffix heap size=141328,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=240 2023-04-12 15:56:22,099 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1681314982089, max timestamp=1681314982096Num uniques 1090; 2023-04-12 15:56:22,099 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=10990, suffix heap size=141328,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=240 2023-04-12 15:56:22,099 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1681314982089, max timestamp=1681314982096Num uniques 1090; 2023-04-12 15:56:22,099 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=10990, suffix heap size=141328,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=240 2023-04-12 15:56:22,125 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1681314982115, max timestamp=1681314982123Num uniques 1099; 2023-04-12 15:56:22,125 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1681314982115, max timestamp=1681314982123Num uniques 1099; 2023-04-12 15:56:22,125 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=11404, suffix heap size=142264,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=249 2023-04-12 15:56:22,125 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=11404, suffix heap size=142264,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=249 2023-04-12 15:56:22,125 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1681314982115, max timestamp=1681314982123Num uniques 1099; 2023-04-12 15:56:22,126 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=11404, suffix heap size=142264,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=249 2023-04-12 15:56:22,153 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1681314982140, max timestamp=1681314982150Num uniques 1108; 2023-04-12 15:56:22,153 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=9520, suffix heap size=143200,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=208 2023-04-12 15:56:22,153 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1681314982140, max timestamp=1681314982150Num uniques 1108; 2023-04-12 15:56:22,153 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=9520, suffix heap size=143200,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=208 2023-04-12 15:56:22,154 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681314982140, max timestamp=1681314982150Num uniques 1108; 2023-04-12 15:56:22,154 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=6870, suffix heap size=143200,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=150 2023-04-12 15:56:22,175 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=267, cellSize=12222, totalHeapSize=28008, min timestamp=1681314982165, max timestamp=1681314982174Num uniques 1067; 2023-04-12 15:56:22,175 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=12222, suffix heap size=138936,new segment heap size=28008 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=267 2023-04-12 15:56:22,176 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=267, cellSize=12222, totalHeapSize=28008, min timestamp=1681314982165, max timestamp=1681314982174Num uniques 1009; 2023-04-12 15:56:22,176 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46212, new segment data size=12222, suffix heap size=132904,new segment heap size=28008 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=267 2023-04-12 15:56:22,176 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=267, cellSize=12222, totalHeapSize=28008, min timestamp=1681314982165, max timestamp=1681314982174Num uniques 1067; 2023-04-12 15:56:22,176 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=12222, suffix heap size=138936,new segment heap size=28008 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=267 2023-04-12 15:56:22,195 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=326, cellSize=14926, totalHeapSize=34144, min timestamp=1681314982186, max timestamp=1681314982193Num uniques 1126; 2023-04-12 15:56:22,195 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=51566, new segment data size=14926, suffix heap size=145072,new segment heap size=34144 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1126, new segment cells count=326 2023-04-12 15:56:22,196 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=326, cellSize=14926, totalHeapSize=34144, min timestamp=1681314982186, max timestamp=1681314982193Num uniques 1126; 2023-04-12 15:56:22,196 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=326, cellSize=14926, totalHeapSize=34144, min timestamp=1681314982186, max timestamp=1681314982193Num uniques 1126; 2023-04-12 15:56:22,196 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=51566, new segment data size=14926, suffix heap size=145072,new segment heap size=34144 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1126, new segment cells count=326 2023-04-12 15:56:22,196 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=51566, new segment data size=14926, suffix heap size=145072,new segment heap size=34144 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1126, new segment cells count=326 2023-04-12 15:56:22,218 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=185, cellSize=8470, totalHeapSize=19480, min timestamp=1681314982209, max timestamp=1681314982216Num uniques 1185; 2023-04-12 15:56:22,219 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=54270, new segment data size=8470, suffix heap size=151208,new segment heap size=19480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1185, new segment cells count=185 2023-04-12 15:56:22,219 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=185, cellSize=8470, totalHeapSize=19480, min timestamp=1681314982209, max timestamp=1681314982216Num uniques 1185; 2023-04-12 15:56:22,219 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=54270, new segment data size=8470, suffix heap size=151208,new segment heap size=19480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1185, new segment cells count=185 2023-04-12 15:56:22,220 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=185, cellSize=8470, totalHeapSize=19480, min timestamp=1681314982209, max timestamp=1681314982216Num uniques 1185; 2023-04-12 15:56:22,220 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=54270, new segment data size=8470, suffix heap size=151208,new segment heap size=19480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1185, new segment cells count=185 2023-04-12 15:56:22,263 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=294, cellSize=13464, totalHeapSize=30816, min timestamp=1681314982254, max timestamp=1681314982261Num uniques 1044; 2023-04-12 15:56:22,263 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47814, new segment data size=13464, suffix heap size=136544,new segment heap size=30816 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1044, new segment cells count=294 2023-04-12 15:56:22,263 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8884, totalHeapSize=20416, min timestamp=1681314982254, max timestamp=1681314982261Num uniques 1044; 2023-04-12 15:56:22,263 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47814, new segment data size=8884, suffix heap size=136544,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1044, new segment cells count=194 2023-04-12 15:56:22,264 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8884, totalHeapSize=20416, min timestamp=1681314982254, max timestamp=1681314982261Num uniques 1044; 2023-04-12 15:56:22,264 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47814, new segment data size=8884, suffix heap size=136544,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1044, new segment cells count=194 2023-04-12 15:56:22,271 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=550, memsize=6.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/A/cd551bfe2ff348fc82480844543cae6e 2023-04-12 15:56:22,289 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=153, cellSize=7005, totalHeapSize=16152, min timestamp=1681314982281, max timestamp=1681314982287Num uniques 1053; 2023-04-12 15:56:22,289 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=153, cellSize=7005, totalHeapSize=16152, min timestamp=1681314982281, max timestamp=1681314982287Num uniques 1053; 2023-04-12 15:56:22,289 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=153, cellSize=7005, totalHeapSize=16152, min timestamp=1681314982281, max timestamp=1681314982287Num uniques 1153; 2023-04-12 15:56:22,290 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48225, new segment data size=7005, suffix heap size=137480,new segment heap size=16152 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1053, new segment cells count=153 2023-04-12 15:56:22,290 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48225, new segment data size=7005, suffix heap size=137480,new segment heap size=16152 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1053, new segment cells count=153 2023-04-12 15:56:22,290 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=52805, new segment data size=7005, suffix heap size=147880,new segment heap size=16152 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1153, new segment cells count=153 2023-04-12 15:56:22,316 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=162, cellSize=7412, totalHeapSize=17088, min timestamp=1681314982304, max timestamp=1681314982313Num uniques 1012; 2023-04-12 15:56:22,316 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=162, cellSize=7412, totalHeapSize=17088, min timestamp=1681314982304, max timestamp=1681314982313Num uniques 1012; 2023-04-12 15:56:22,316 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46342, new segment data size=7412, suffix heap size=133216,new segment heap size=17088 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1012, new segment cells count=162 2023-04-12 15:56:22,316 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=212, cellSize=9702, totalHeapSize=22288, min timestamp=1681314982304, max timestamp=1681314982313Num uniques 1012; 2023-04-12 15:56:22,316 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46342, new segment data size=7412, suffix heap size=133216,new segment heap size=17088 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1012, new segment cells count=162 2023-04-12 15:56:22,316 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46342, new segment data size=9702, suffix heap size=133216,new segment heap size=22288 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1012, new segment cells count=212 2023-04-12 15:56:22,338 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=221, cellSize=10116, totalHeapSize=23224, min timestamp=1681314982323, max timestamp=1681314982335Num uniques 1071; 2023-04-12 15:56:22,338 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=221, cellSize=10116, totalHeapSize=23224, min timestamp=1681314982323, max timestamp=1681314982335Num uniques 1021; 2023-04-12 15:56:22,338 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=221, cellSize=10116, totalHeapSize=23224, min timestamp=1681314982323, max timestamp=1681314982335Num uniques 1021; 2023-04-12 15:56:22,338 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46756, new segment data size=10116, suffix heap size=134152,new segment heap size=23224 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1021, new segment cells count=221 2023-04-12 15:56:22,338 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49046, new segment data size=10116, suffix heap size=139352,new segment heap size=23224 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1071, new segment cells count=221 2023-04-12 15:56:22,338 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46756, new segment data size=10116, suffix heap size=134152,new segment heap size=23224 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1021, new segment cells count=221 2023-04-12 15:56:22,362 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=230, cellSize=10530, totalHeapSize=24160, min timestamp=1681314982348, max timestamp=1681314982359Num uniques 1080; 2023-04-12 15:56:22,362 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=230, cellSize=10530, totalHeapSize=24160, min timestamp=1681314982348, max timestamp=1681314982359Num uniques 1080; 2023-04-12 15:56:22,362 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49460, new segment data size=10530, suffix heap size=140288,new segment heap size=24160 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1080, new segment cells count=230 2023-04-12 15:56:22,362 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49460, new segment data size=10530, suffix heap size=140288,new segment heap size=24160 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1080, new segment cells count=230 2023-04-12 15:56:22,362 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=230, cellSize=10530, totalHeapSize=24160, min timestamp=1681314982348, max timestamp=1681314982359Num uniques 1080; 2023-04-12 15:56:22,362 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49460, new segment data size=10530, suffix heap size=140288,new segment heap size=24160 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1080, new segment cells count=230 2023-04-12 15:56:22,384 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=189, cellSize=8644, totalHeapSize=19896, min timestamp=1681314982373, max timestamp=1681314982383Num uniques 1089; 2023-04-12 15:56:22,385 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49864, new segment data size=8644, suffix heap size=141224,new segment heap size=19896 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1089, new segment cells count=189 2023-04-12 15:56:22,385 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=189, cellSize=8654, totalHeapSize=19896, min timestamp=1681314982373, max timestamp=1681314982381Num uniques 1089; 2023-04-12 15:56:22,385 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=189, cellSize=8654, totalHeapSize=19896, min timestamp=1681314982373, max timestamp=1681314982381Num uniques 1089; 2023-04-12 15:56:22,385 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49874, new segment data size=8654, suffix heap size=141224,new segment heap size=19896 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1089, new segment cells count=189 2023-04-12 15:56:22,385 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49874, new segment data size=8654, suffix heap size=141224,new segment heap size=19896 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1089, new segment cells count=189 2023-04-12 15:56:22,430 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=298, cellSize=13648, totalHeapSize=31232, min timestamp=1681314982413, max timestamp=1681314982428Num uniques 1048; 2023-04-12 15:56:22,431 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47998, new segment data size=13648, suffix heap size=136960,new segment heap size=31232 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1048, new segment cells count=298 2023-04-12 15:56:22,431 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=298, cellSize=13648, totalHeapSize=31232, min timestamp=1681314982413, max timestamp=1681314982428Num uniques 1048; 2023-04-12 15:56:22,431 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=298, cellSize=13648, totalHeapSize=31232, min timestamp=1681314982413, max timestamp=1681314982428Num uniques 1048; 2023-04-12 15:56:22,431 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47998, new segment data size=13648, suffix heap size=136960,new segment heap size=31232 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1048, new segment cells count=298 2023-04-12 15:56:22,431 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47998, new segment data size=13648, suffix heap size=136960,new segment heap size=31232 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1048, new segment cells count=298 2023-04-12 15:56:22,453 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=257, cellSize=11765, totalHeapSize=26968, min timestamp=1681314982445, max timestamp=1681314982451Num uniques 1157; 2023-04-12 15:56:22,453 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=52985, new segment data size=11765, suffix heap size=148296,new segment heap size=26968 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1157, new segment cells count=257 2023-04-12 15:56:22,453 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=257, cellSize=11765, totalHeapSize=26968, min timestamp=1681314982445, max timestamp=1681314982451Num uniques 1157; 2023-04-12 15:56:22,453 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=257, cellSize=11765, totalHeapSize=26968, min timestamp=1681314982445, max timestamp=1681314982451Num uniques 1157; 2023-04-12 15:56:22,453 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=52985, new segment data size=11765, suffix heap size=148296,new segment heap size=26968 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1157, new segment cells count=257 2023-04-12 15:56:22,454 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=52985, new segment data size=11765, suffix heap size=148296,new segment heap size=26968 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1157, new segment cells count=257 2023-04-12 15:56:22,479 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=166, cellSize=7596, totalHeapSize=17504, min timestamp=1681314982463, max timestamp=1681314982476Num uniques 1116; 2023-04-12 15:56:22,479 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=166, cellSize=7596, totalHeapSize=17504, min timestamp=1681314982463, max timestamp=1681314982476Num uniques 1116; 2023-04-12 15:56:22,479 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=51106, new segment data size=7596, suffix heap size=144032,new segment heap size=17504 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1116, new segment cells count=166 2023-04-12 15:56:22,479 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=166, cellSize=7596, totalHeapSize=17504, min timestamp=1681314982463, max timestamp=1681314982476Num uniques 1116; 2023-04-12 15:56:22,479 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=51106, new segment data size=7596, suffix heap size=144032,new segment heap size=17504 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1116, new segment cells count=166 2023-04-12 15:56:22,479 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=51106, new segment data size=7596, suffix heap size=144032,new segment heap size=17504 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1116, new segment cells count=166 2023-04-12 15:56:22,503 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=175, cellSize=8010, totalHeapSize=18440, min timestamp=1681314982494, max timestamp=1681314982500Num uniques 1025; 2023-04-12 15:56:22,503 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46940, new segment data size=8010, suffix heap size=134568,new segment heap size=18440 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1025, new segment cells count=175 2023-04-12 15:56:22,503 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=175, cellSize=8010, totalHeapSize=18440, min timestamp=1681314982494, max timestamp=1681314982500Num uniques 1025; 2023-04-12 15:56:22,503 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46940, new segment data size=8010, suffix heap size=134568,new segment heap size=18440 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1025, new segment cells count=175 2023-04-12 15:56:22,504 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=175, cellSize=8010, totalHeapSize=18440, min timestamp=1681314982494, max timestamp=1681314982500Num uniques 1025; 2023-04-12 15:56:22,504 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46940, new segment data size=8010, suffix heap size=134568,new segment heap size=18440 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1025, new segment cells count=175 2023-04-12 15:56:22,535 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=234, cellSize=10714, totalHeapSize=24576, min timestamp=1681314982514, max timestamp=1681314982532Num uniques 1034; 2023-04-12 15:56:22,535 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=234, cellSize=10714, totalHeapSize=24576, min timestamp=1681314982514, max timestamp=1681314982532Num uniques 1034; 2023-04-12 15:56:22,535 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47354, new segment data size=10714, suffix heap size=135504,new segment heap size=24576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1034, new segment cells count=234 2023-04-12 15:56:22,535 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=234, cellSize=10714, totalHeapSize=24576, min timestamp=1681314982514, max timestamp=1681314982532Num uniques 1034; 2023-04-12 15:56:22,535 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47354, new segment data size=10714, suffix heap size=135504,new segment heap size=24576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1034, new segment cells count=234 2023-04-12 15:56:22,535 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47354, new segment data size=10714, suffix heap size=135504,new segment heap size=24576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1034, new segment cells count=234 2023-04-12 15:56:22,583 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=193, cellSize=8838, totalHeapSize=20312, min timestamp=1681314982577, max timestamp=1681314982581Num uniques 1093; 2023-04-12 15:56:22,583 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=50058, new segment data size=8838, suffix heap size=141640,new segment heap size=20312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1093, new segment cells count=193 2023-04-12 15:56:22,583 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=193, cellSize=8838, totalHeapSize=20312, min timestamp=1681314982577, max timestamp=1681314982581Num uniques 1093; 2023-04-12 15:56:22,584 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=50058, new segment data size=8838, suffix heap size=141640,new segment heap size=20312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1093, new segment cells count=193 2023-04-12 15:56:22,584 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=193, cellSize=8838, totalHeapSize=20312, min timestamp=1681314982577, max timestamp=1681314982581Num uniques 1093; 2023-04-12 15:56:22,584 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50058, new segment data size=8838, suffix heap size=141640,new segment heap size=20312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1093, new segment cells count=193 2023-04-12 15:56:22,608 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=202, cellSize=9250, totalHeapSize=21248, min timestamp=1681314982596, max timestamp=1681314982605Num uniques 1052; 2023-04-12 15:56:22,608 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48180, new segment data size=9250, suffix heap size=137376,new segment heap size=21248 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1052, new segment cells count=202 2023-04-12 15:56:22,608 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=202, cellSize=9250, totalHeapSize=21248, min timestamp=1681314982596, max timestamp=1681314982605Num uniques 1052; 2023-04-12 15:56:22,608 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=202, cellSize=9250, totalHeapSize=21248, min timestamp=1681314982596, max timestamp=1681314982605Num uniques 1052; 2023-04-12 15:56:22,608 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48180, new segment data size=9250, suffix heap size=137376,new segment heap size=21248 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1052, new segment cells count=202 2023-04-12 15:56:22,608 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48180, new segment data size=9250, suffix heap size=137376,new segment heap size=21248 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1052, new segment cells count=202 2023-04-12 15:56:22,633 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=161, cellSize=7366, totalHeapSize=16984, min timestamp=1681314982626, max timestamp=1681314982631Num uniques 1061; 2023-04-12 15:56:22,633 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48586, new segment data size=7366, suffix heap size=138312,new segment heap size=16984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1061, new segment cells count=161 2023-04-12 15:56:22,633 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=161, cellSize=7366, totalHeapSize=16984, min timestamp=1681314982626, max timestamp=1681314982631Num uniques 1061; 2023-04-12 15:56:22,633 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=161, cellSize=7366, totalHeapSize=16984, min timestamp=1681314982626, max timestamp=1681314982631Num uniques 1061; 2023-04-12 15:56:22,633 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48586, new segment data size=7366, suffix heap size=138312,new segment heap size=16984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1061, new segment cells count=161 2023-04-12 15:56:22,634 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48586, new segment data size=7366, suffix heap size=138312,new segment heap size=16984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1061, new segment cells count=161 2023-04-12 15:56:22,657 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=220, cellSize=10070, totalHeapSize=23120, min timestamp=1681314982649, max timestamp=1681314982655Num uniques 1020; 2023-04-12 15:56:22,657 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46710, new segment data size=10070, suffix heap size=134048,new segment heap size=23120 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1020, new segment cells count=220 2023-04-12 15:56:22,657 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=220, cellSize=10070, totalHeapSize=23120, min timestamp=1681314982649, max timestamp=1681314982655Num uniques 1020; 2023-04-12 15:56:22,657 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=220, cellSize=10070, totalHeapSize=23120, min timestamp=1681314982649, max timestamp=1681314982655Num uniques 1020; 2023-04-12 15:56:22,658 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46710, new segment data size=10070, suffix heap size=134048,new segment heap size=23120 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1020, new segment cells count=220 2023-04-12 15:56:22,658 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46710, new segment data size=10070, suffix heap size=134048,new segment heap size=23120 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1020, new segment cells count=220 2023-04-12 15:56:22,680 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=179, cellSize=8194, totalHeapSize=18856, min timestamp=1681314982662, max timestamp=1681314982678Num uniques 1079; 2023-04-12 15:56:22,680 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49414, new segment data size=8194, suffix heap size=140184,new segment heap size=18856 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1079, new segment cells count=179 2023-04-12 15:56:22,680 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=179, cellSize=8194, totalHeapSize=18856, min timestamp=1681314982662, max timestamp=1681314982678Num uniques 1079; 2023-04-12 15:56:22,680 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=179, cellSize=8194, totalHeapSize=18856, min timestamp=1681314982662, max timestamp=1681314982678Num uniques 1079; 2023-04-12 15:56:22,680 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49414, new segment data size=8194, suffix heap size=140184,new segment heap size=18856 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1079, new segment cells count=179 2023-04-12 15:56:22,680 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49414, new segment data size=8194, suffix heap size=140184,new segment heap size=18856 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1079, new segment cells count=179 2023-04-12 15:56:22,682 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=550 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/B/4b016300f9b44297a4e8254f7b738dc4 2023-04-12 15:56:22,697 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=550 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/C/268d259a4b9042f096586441a2cb9785 2023-04-12 15:56:22,701 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/A/cd551bfe2ff348fc82480844543cae6e as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/cd551bfe2ff348fc82480844543cae6e 2023-04-12 15:56:22,704 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/cd551bfe2ff348fc82480844543cae6e, entries=150, sequenceid=550, filesize=25.4 K 2023-04-12 15:56:22,705 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/B/4b016300f9b44297a4e8254f7b738dc4 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/4b016300f9b44297a4e8254f7b738dc4 2023-04-12 15:56:22,709 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/4b016300f9b44297a4e8254f7b738dc4, entries=100, sequenceid=550, filesize=9.5 K 2023-04-12 15:56:22,710 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/C/268d259a4b9042f096586441a2cb9785 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/268d259a4b9042f096586441a2cb9785 2023-04-12 15:56:22,714 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=188, cellSize=8608, totalHeapSize=19792, min timestamp=1681314982705, max timestamp=1681314982711Num uniques 1038; 2023-04-12 15:56:22,714 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=188, cellSize=8608, totalHeapSize=19792, min timestamp=1681314982705, max timestamp=1681314982711Num uniques 1038; 2023-04-12 15:56:22,714 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=188, cellSize=8608, totalHeapSize=19792, min timestamp=1681314982705, max timestamp=1681314982711Num uniques 1038; 2023-04-12 15:56:22,714 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47538, new segment data size=8608, suffix heap size=135920,new segment heap size=19792 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1038, new segment cells count=188 2023-04-12 15:56:22,714 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47538, new segment data size=8608, suffix heap size=135920,new segment heap size=19792 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1038, new segment cells count=188 2023-04-12 15:56:22,714 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47538, new segment data size=8608, suffix heap size=135920,new segment heap size=19792 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1038, new segment cells count=188 2023-04-12 15:56:22,716 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/268d259a4b9042f096586441a2cb9785, entries=100, sequenceid=550, filesize=9.5 K 2023-04-12 15:56:22,717 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=33.54 KB/34350 for 4870003209f366bc50ef2084e6e2cf03 in 888ms, sequenceid=550, compaction requested=true 2023-04-12 15:56:22,717 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4870003209f366bc50ef2084e6e2cf03: 2023-04-12 15:56:22,717 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:22,717 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:56:22,717 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:56:22,717 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:22,717 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:56:22,717 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:56:22,718 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:22,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:22,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:22,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:22,719 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:22,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:22,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:22,719 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:22,719 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:56:22,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:22,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:22,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:22,720 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:56:22,721 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@322bfd6d[Count = 0] remaining members to acquire global barrier 2023-04-12 15:56:22,721 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:56:22,721 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:22,721 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:22,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:22,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:22,722 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:56:22,722 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:56:22,722 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:56:22,722 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:22,722 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:56:22,724 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:22,724 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:56:22,724 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:22,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:22,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:22,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:22,724 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:56:22,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:22,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:22,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:22,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:22,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:22,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:22,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:22,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:56:22,726 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:56:22,726 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:56:22,726 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:56:22,726 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:56:22,726 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:56:22,726 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:56:22,728 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:22,728 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:22,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:22,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:22,728 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:22,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:22,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:22,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:22,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:22,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:22,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:22,728 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:22,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:22,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:22,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:22,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:22,729 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:22,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:22,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:22,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:22,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:22,731 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:22,731 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:22,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:22,731 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:56:22,731 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:56:22,731 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:56:22,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:22,731 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:22,731 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:22,731 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:22,731 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:22,731 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:22,731 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:22,732 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:22,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:22,732 DEBUG [Thread-4110] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:56:22,732 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:22,732 DEBUG [Thread-4110] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:56:22,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:22,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on 4870003209f366bc50ef2084e6e2cf03 2023-04-12 15:56:22,758 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4870003209f366bc50ef2084e6e2cf03 3/3 column families, dataSize=60.38 KB heapSize=163.13 KB 2023-04-12 15:56:22,759 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=A 2023-04-12 15:56:22,759 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:56:22,759 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=B 2023-04-12 15:56:22,759 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:56:22,759 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=C 2023-04-12 15:56:22,759 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:56:22,773 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230412ac035439b4984e17b8e2da243883b2dd to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412ac035439b4984e17b8e2da243883b2dd 2023-04-12 15:56:22,781 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1108, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/A/60a60b1639414c1da097244c75f7273e 2023-04-12 15:56:22,791 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681314982776, max timestamp=1681314982789Num uniques 859; 2023-04-12 15:56:22,791 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681314982776, max timestamp=1681314982789Num uniques 859; 2023-04-12 15:56:22,791 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-12 15:56:22,791 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681314982776, max timestamp=1681314982789Num uniques 859; 2023-04-12 15:56:22,791 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-12 15:56:22,791 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-12 15:56:22,815 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:22,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 400 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53360 deadline: 1681315042811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:22,815 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:22,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 414 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53324 deadline: 1681315042812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:22,819 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:22,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 407 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53392 deadline: 1681315042815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:22,832 DEBUG [Thread-4110] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:56:22,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:56:22,834 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:56:22,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:56:22,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:56:22,835 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:56:22,835 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:22,835 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:56:22,835 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:56:22,835 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:22,835 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:22,836 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:22,836 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:22,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:22,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:22,837 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:22,837 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:56:22,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:22,837 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:22,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:56:22,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:22,837 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:56:22,838 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:56:22,838 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:56:22,838 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:56:22,838 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:22,838 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:56:22,838 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:56:22,838 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:56:22,838 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:22,838 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. started... 2023-04-12 15:56:22,838 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. as already flushing 2023-04-12 15:56:22,838 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:22,838 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:56:22,838 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:56:22,838 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:22,839 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:56:22,839 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:56:22,841 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:22,841 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:22,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:22,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:22,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:22,842 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:22,842 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:56:22,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:22,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:22,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:22,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:22,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:22,843 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:56:22,843 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@964a8ae[Count = 0] remaining members to acquire global barrier 2023-04-12 15:56:22,843 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:56:22,843 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:22,844 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:22,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:22,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:22,844 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:56:22,844 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:56:22,845 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:56:22,845 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:22,845 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:56:22,846 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:56:22,846 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:22,846 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:22,846 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:56:22,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:22,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:22,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:22,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:22,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:22,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:22,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:22,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:22,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:22,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:22,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:56:22,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:56:22,849 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:56:22,849 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:56:22,849 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:56:22,849 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:56:22,849 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:56:22,850 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:22,850 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:22,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:22,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:22,850 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:22,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:22,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:22,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:22,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:22,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:22,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:22,851 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:22,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:22,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:22,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:22,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:22,853 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:22,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:22,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:22,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:22,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:22,857 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:22,857 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:22,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:22,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:22,857 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:22,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:56:22,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:56:22,857 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:22,857 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:22,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:22,857 DEBUG [Thread-4110] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:56:22,857 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:22,857 DEBUG [Thread-4110] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:56:22,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:22,857 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:56:22,857 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:22,858 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:22,858 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:22,858 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:22,917 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:22,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 402 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53360 deadline: 1681315042916, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:22,917 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:22,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 416 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53324 deadline: 1681315042916, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:22,922 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:22,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 409 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53392 deadline: 1681315042920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:22,957 DEBUG [Thread-4110] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:56:22,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:56:22,959 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:56:22,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:56:22,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:56:22,961 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:56:22,961 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:22,961 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:56:22,961 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:56:22,961 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:22,961 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:22,962 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:22,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:22,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:22,962 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:22,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:22,963 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:22,963 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:56:22,963 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:22,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:56:22,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:22,963 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:56:22,963 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:56:22,963 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:56:22,964 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:56:22,964 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:22,964 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:56:22,964 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:56:22,964 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:56:22,964 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:22,964 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. started... 2023-04-12 15:56:22,964 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. as already flushing 2023-04-12 15:56:22,964 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:22,964 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:56:22,964 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:56:22,964 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:22,964 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:56:22,964 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:56:22,966 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:22,966 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:22,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:22,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:22,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:22,966 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:22,966 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:56:22,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:22,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:22,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:22,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:22,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:22,967 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:56:22,967 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@170b9baf[Count = 0] remaining members to acquire global barrier 2023-04-12 15:56:22,967 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:56:22,968 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:22,969 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:22,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:22,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:22,969 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:56:22,969 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:56:22,969 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:56:22,969 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:22,969 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:56:22,971 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:56:22,971 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:22,971 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:22,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:22,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:22,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:22,971 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:56:22,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:22,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:22,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:22,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:22,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:22,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:22,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:22,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:56:22,974 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:56:22,974 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:56:22,974 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:56:22,974 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:56:22,974 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:56:22,974 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:56:22,975 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:22,975 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:22,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:22,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:22,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:22,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:22,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:22,975 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:22,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:22,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:22,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:22,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:22,976 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:22,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:22,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:22,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:22,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:22,978 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:22,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:22,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:22,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:22,980 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:22,980 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:22,980 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:22,980 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:22,980 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:22,980 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:22,980 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:22,980 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:22,980 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:22,980 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:22,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:22,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:22,980 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:56:22,980 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:56:22,980 DEBUG [Thread-4110] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:56:22,981 DEBUG [Thread-4110] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:56:22,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:22,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:23,081 DEBUG [Thread-4110] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:56:23,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:56:23,083 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:56:23,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:56:23,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:56:23,084 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:56:23,084 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:23,084 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:56:23,084 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:56:23,084 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:23,084 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:23,085 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:23,085 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:23,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:23,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:23,086 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:23,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:23,086 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:56:23,086 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:23,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:56:23,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:23,086 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:56:23,087 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:56:23,087 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:56:23,087 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:23,087 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:56:23,087 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:56:23,087 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:56:23,087 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:23,087 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. started... 2023-04-12 15:56:23,087 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. as already flushing 2023-04-12 15:56:23,087 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:23,087 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:56:23,087 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:56:23,087 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:23,087 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:56:23,087 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:56:23,089 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:23,089 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:23,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:23,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:23,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:23,090 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:23,090 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:56:23,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:23,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:23,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:23,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:23,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:23,091 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:56:23,091 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51f1d645[Count = 0] remaining members to acquire global barrier 2023-04-12 15:56:23,091 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:56:23,091 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:23,095 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:23,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:23,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:23,095 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:56:23,095 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:23,095 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:56:23,095 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:56:23,095 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:56:23,096 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:56:23,096 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:23,096 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:56:23,096 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:23,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:23,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:23,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:23,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:23,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:23,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:23,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:23,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:23,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:23,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:23,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:56:23,099 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:56:23,099 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:56:23,099 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:56:23,099 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:56:23,099 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:56:23,099 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:56:23,100 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:23,100 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:23,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:23,100 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:23,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:23,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:23,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:23,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:23,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:23,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:23,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:23,101 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:23,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:23,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:23,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:23,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:23,102 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:23,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:23,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:23,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:23,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:23,105 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:23,105 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:23,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:23,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:23,105 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:23,105 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:23,105 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:23,105 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:23,105 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:23,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:23,105 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:23,106 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:23,105 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:56:23,105 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:56:23,106 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:56:23,105 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:23,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:23,106 DEBUG [Thread-4110] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:56:23,106 DEBUG [Thread-4110] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:56:23,122 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:23,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 404 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53360 deadline: 1681315043118, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:23,123 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:23,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 418 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53324 deadline: 1681315043119, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:23,126 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:23,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 411 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53392 deadline: 1681315043123, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:23,190 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=1108 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/B/8da03c5f051c411198dde322a4b8a86a 2023-04-12 15:56:23,206 DEBUG [Thread-4110] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:56:23,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:56:23,209 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:56:23,210 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=1108 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/C/0d37ec0e606942b28ed9852568edc41e 2023-04-12 15:56:23,215 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/A/60a60b1639414c1da097244c75f7273e as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/60a60b1639414c1da097244c75f7273e 2023-04-12 15:56:23,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:56:23,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:56:23,216 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:56:23,217 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:23,217 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:56:23,217 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:56:23,217 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:23,217 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:23,218 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:23,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:23,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:23,218 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:23,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:23,219 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:23,219 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:56:23,219 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:23,219 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/60a60b1639414c1da097244c75f7273e, entries=150, sequenceid=1108, filesize=25.4 K 2023-04-12 15:56:23,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:56:23,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:23,220 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:56:23,220 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:56:23,220 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:56:23,220 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:56:23,220 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:23,220 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:56:23,220 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:56:23,220 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:56:23,220 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/B/8da03c5f051c411198dde322a4b8a86a as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/8da03c5f051c411198dde322a4b8a86a 2023-04-12 15:56:23,220 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:23,220 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. started... 2023-04-12 15:56:23,221 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. as already flushing 2023-04-12 15:56:23,221 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:23,221 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:56:23,221 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:56:23,221 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:23,221 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:56:23,221 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:56:23,222 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:23,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:23,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:23,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:23,222 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:23,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:23,223 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:23,223 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:56:23,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:23,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:23,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:23,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:23,224 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/8da03c5f051c411198dde322a4b8a86a, entries=150, sequenceid=1108, filesize=11.9 K 2023-04-12 15:56:23,224 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:56:23,224 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c5c9983[Count = 0] remaining members to acquire global barrier 2023-04-12 15:56:23,224 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:56:23,224 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:23,225 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/C/0d37ec0e606942b28ed9852568edc41e as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/0d37ec0e606942b28ed9852568edc41e 2023-04-12 15:56:23,225 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:23,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:23,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:23,225 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:23,226 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:56:23,225 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:56:23,226 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:56:23,226 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:56:23,228 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:23,228 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:56:23,228 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:23,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:23,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:23,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:23,228 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:56:23,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:23,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:23,229 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/0d37ec0e606942b28ed9852568edc41e, entries=150, sequenceid=1108, filesize=11.9 K 2023-04-12 15:56:23,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:23,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:23,230 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~163.08 KB/166992, currentSize=127.47 KB/130530 for 4870003209f366bc50ef2084e6e2cf03 in 472ms, sequenceid=1108, compaction requested=true 2023-04-12 15:56:23,230 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4870003209f366bc50ef2084e6e2cf03: 2023-04-12 15:56:23,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:23,230 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:56:23,230 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-12 15:56:23,230 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-12 15:56:23,230 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:56:23,230 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:56:23,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:23,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:23,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:56:23,231 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:56:23,231 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:56:23,231 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:56:23,231 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:56:23,231 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:56:23,231 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:56:23,232 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 131316 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-12 15:56:23,232 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59610 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-12 15:56:23,232 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1912): 4870003209f366bc50ef2084e6e2cf03/A is initiating minor compaction (all files) 2023-04-12 15:56:23,232 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1912): 4870003209f366bc50ef2084e6e2cf03/B is initiating minor compaction (all files) 2023-04-12 15:56:23,232 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4870003209f366bc50ef2084e6e2cf03/A in TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:23,232 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4870003209f366bc50ef2084e6e2cf03/B in TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:23,232 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/2d948b679bd44a318a5eb0b0558b9384, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/77e3e05f931241509e187715a572dc97, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/36d9863e74c44fd9858afafe8a72e980, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/cd551bfe2ff348fc82480844543cae6e, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/60a60b1639414c1da097244c75f7273e] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp, totalSize=128.2 K 2023-04-12 15:56:23,232 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/474092d4fe3142ec986b7887effde366, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/6e5f54175c004e44bb5e5ee928cdf2c4, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/e67caf640b0545e1a783f81b3655d89a, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/4b016300f9b44297a4e8254f7b738dc4, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/8da03c5f051c411198dde322a4b8a86a] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp, totalSize=58.2 K 2023-04-12 15:56:23,232 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 2d948b679bd44a318a5eb0b0558b9384, keycount=150, bloomtype=ROW, size=26.5 K, encoding=NONE, compression=NONE, seqNum=509, earliestPutTs=1681314979727 2023-04-12 15:56:23,233 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 474092d4fe3142ec986b7887effde366, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=509, earliestPutTs=1681314979727 2023-04-12 15:56:23,233 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 77e3e05f931241509e187715a572dc97, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=527, earliestPutTs=1681314980055 2023-04-12 15:56:23,233 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 6e5f54175c004e44bb5e5ee928cdf2c4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=527, earliestPutTs=1681314980055 2023-04-12 15:56:23,233 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 36d9863e74c44fd9858afafe8a72e980, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=544, earliestPutTs=1681314980686 2023-04-12 15:56:23,233 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:23,233 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting cd551bfe2ff348fc82480844543cae6e, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=550, earliestPutTs=1681314981825 2023-04-12 15:56:23,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:23,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:23,233 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:23,233 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting e67caf640b0545e1a783f81b3655d89a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=544, earliestPutTs=1681314980686 2023-04-12 15:56:23,233 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:23,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:23,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:23,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:23,234 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 60a60b1639414c1da097244c75f7273e, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=1108, earliestPutTs=1681314982707 2023-04-12 15:56:23,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:23,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:23,234 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 4b016300f9b44297a4e8254f7b738dc4, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=550, earliestPutTs=1681314981826 2023-04-12 15:56:23,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:23,234 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:23,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:23,235 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 8da03c5f051c411198dde322a4b8a86a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1108, earliestPutTs=1681314982707 2023-04-12 15:56:23,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:23,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:23,236 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:23,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:23,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:23,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:23,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:23,238 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:23,238 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:23,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:23,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:23,238 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:23,238 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:23,238 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:23,239 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:23,239 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:23,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:56:23,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:56:23,239 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:23,239 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:23,239 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:23,239 DEBUG [Thread-4110] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:56:23,239 DEBUG [Thread-4110] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:56:23,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:23,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:23,239 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:56:23,244 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] throttle.PressureAwareThroughputController(145): 4870003209f366bc50ef2084e6e2cf03#B#compaction#666 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:56:23,249 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4870003209f366bc50ef2084e6e2cf03#A#compaction#665 average throughput is 2.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:56:23,263 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/B/139a72300c38492c9fb4159827dea1c6 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/139a72300c38492c9fb4159827dea1c6 2023-04-12 15:56:23,269 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 4870003209f366bc50ef2084e6e2cf03/B of 4870003209f366bc50ef2084e6e2cf03 into 139a72300c38492c9fb4159827dea1c6(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:56:23,269 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4870003209f366bc50ef2084e6e2cf03: 2023-04-12 15:56:23,269 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03., storeName=4870003209f366bc50ef2084e6e2cf03/B, priority=11, startTime=1681314983230; duration=0sec 2023-04-12 15:56:23,269 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:56:23,269 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-12 15:56:23,271 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59576 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-12 15:56:23,271 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1912): 4870003209f366bc50ef2084e6e2cf03/C is initiating minor compaction (all files) 2023-04-12 15:56:23,271 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4870003209f366bc50ef2084e6e2cf03/C in TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:23,271 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/5218664995954f6a97811f3158dc1bb7, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/df3761fa0d3c4398bc41dd3ff9626f86, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/663db4c933884290a21af172f7a34a3a, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/268d259a4b9042f096586441a2cb9785, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/0d37ec0e606942b28ed9852568edc41e] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp, totalSize=58.2 K 2023-04-12 15:56:23,272 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 5218664995954f6a97811f3158dc1bb7, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=509, earliestPutTs=1681314979727 2023-04-12 15:56:23,272 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting df3761fa0d3c4398bc41dd3ff9626f86, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=527, earliestPutTs=1681314980055 2023-04-12 15:56:23,272 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/A/11f9a24bd9ea4268881aaf2d5e046a6b as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/11f9a24bd9ea4268881aaf2d5e046a6b 2023-04-12 15:56:23,272 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 663db4c933884290a21af172f7a34a3a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=544, earliestPutTs=1681314980686 2023-04-12 15:56:23,273 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 268d259a4b9042f096586441a2cb9785, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=550, earliestPutTs=1681314981826 2023-04-12 15:56:23,273 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 0d37ec0e606942b28ed9852568edc41e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1108, earliestPutTs=1681314982707 2023-04-12 15:56:23,277 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 4870003209f366bc50ef2084e6e2cf03/A of 4870003209f366bc50ef2084e6e2cf03 into 11f9a24bd9ea4268881aaf2d5e046a6b(size=26.7 K), total size for store is 26.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:56:23,277 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4870003209f366bc50ef2084e6e2cf03: 2023-04-12 15:56:23,277 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03., storeName=4870003209f366bc50ef2084e6e2cf03/A, priority=11, startTime=1681314983230; duration=0sec 2023-04-12 15:56:23,277 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:56:23,282 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] throttle.PressureAwareThroughputController(145): 4870003209f366bc50ef2084e6e2cf03#C#compaction#667 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:56:23,297 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/C/81a3c88580514d57a41154eaa484c309 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/81a3c88580514d57a41154eaa484c309 2023-04-12 15:56:23,304 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 4870003209f366bc50ef2084e6e2cf03/C of 4870003209f366bc50ef2084e6e2cf03 into 81a3c88580514d57a41154eaa484c309(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:56:23,304 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4870003209f366bc50ef2084e6e2cf03: 2023-04-12 15:56:23,304 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03., storeName=4870003209f366bc50ef2084e6e2cf03/C, priority=11, startTime=1681314983230; duration=0sec 2023-04-12 15:56:23,304 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:56:23,339 DEBUG [Thread-4110] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:56:23,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:56:23,341 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:56:23,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:56:23,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:56:23,342 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:56:23,342 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:23,342 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:56:23,342 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:56:23,343 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:23,343 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:23,344 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:23,344 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:23,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:23,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:23,344 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:23,344 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:56:23,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:23,345 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:23,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:56:23,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:23,345 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:56:23,345 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:56:23,346 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:56:23,346 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:56:23,346 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:23,346 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:56:23,348 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:56:23,348 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:56:23,348 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:23,348 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. started... 2023-04-12 15:56:23,348 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4870003209f366bc50ef2084e6e2cf03 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-12 15:56:23,348 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=A 2023-04-12 15:56:23,348 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:56:23,349 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=B 2023-04-12 15:56:23,349 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:56:23,349 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=C 2023-04-12 15:56:23,349 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 15:56:23,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on 4870003209f366bc50ef2084e6e2cf03 2023-04-12 15:56:23,427 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. as already flushing 2023-04-12 15:56:23,436 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:23,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 423 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53324 deadline: 1681315043434, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:23,438 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:23,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 415 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53392 deadline: 1681315043435, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:23,438 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:23,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 410 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53360 deadline: 1681315043435, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:23,538 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:23,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 425 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53324 deadline: 1681315043538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:23,541 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:23,541 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:23,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 417 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53392 deadline: 1681315043539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:23,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 412 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53360 deadline: 1681315043539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:23,743 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:23,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 427 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53324 deadline: 1681315043741, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:23,744 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:23,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 414 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53360 deadline: 1681315043742, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:23,744 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:23,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 419 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53392 deadline: 1681315043743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:23,770 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304120fdd289d5907412188710244e845b64c to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304120fdd289d5907412188710244e845b64c 2023-04-12 15:56:23,779 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1146, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/A/211b68e577744284a3580da1d4f6ce35 2023-04-12 15:56:23,788 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=1146 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/B/d429a62d40b34216ba046da40ebffff0 2023-04-12 15:56:23,797 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=1146 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/C/5a07130d038d402e8777cf9df6fe0a8f 2023-04-12 15:56:23,801 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/A/211b68e577744284a3580da1d4f6ce35 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/211b68e577744284a3580da1d4f6ce35 2023-04-12 15:56:23,804 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/211b68e577744284a3580da1d4f6ce35, entries=150, sequenceid=1146, filesize=25.4 K 2023-04-12 15:56:23,805 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/B/d429a62d40b34216ba046da40ebffff0 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/d429a62d40b34216ba046da40ebffff0 2023-04-12 15:56:23,808 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/d429a62d40b34216ba046da40ebffff0, entries=150, sequenceid=1146, filesize=11.9 K 2023-04-12 15:56:23,809 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/C/5a07130d038d402e8777cf9df6fe0a8f as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/5a07130d038d402e8777cf9df6fe0a8f 2023-04-12 15:56:23,813 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/5a07130d038d402e8777cf9df6fe0a8f, entries=150, sequenceid=1146, filesize=11.9 K 2023-04-12 15:56:23,815 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=60.38 KB/61830 for 4870003209f366bc50ef2084e6e2cf03 in 467ms, sequenceid=1146, compaction requested=false 2023-04-12 15:56:23,815 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4870003209f366bc50ef2084e6e2cf03: 2023-04-12 15:56:23,815 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:23,815 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:56:23,815 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:56:23,815 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:23,816 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:56:23,816 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:56:23,818 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:23,818 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:23,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:23,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:23,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:23,819 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:23,819 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:56:23,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:23,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:23,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:23,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:23,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:23,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:56:23,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10d1777e[Count = 0] remaining members to acquire global barrier 2023-04-12 15:56:23,821 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:56:23,821 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:23,822 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:23,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:23,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:23,822 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:56:23,822 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:56:23,822 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:56:23,823 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:23,823 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:56:23,823 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:23,823 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:56:23,823 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:23,824 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:56:23,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:23,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:23,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:23,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:23,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:23,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:23,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:23,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:23,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:23,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:23,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:56:23,827 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:56:23,827 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:56:23,827 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:56:23,827 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:56:23,827 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:56:23,828 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:56:23,830 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:23,830 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:23,830 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:23,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:23,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:23,831 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:23,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:23,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:23,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:23,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:23,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:23,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:23,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:23,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:23,832 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:23,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:23,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:23,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:23,835 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:23,835 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:23,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:23,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:23,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:23,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:23,835 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:23,835 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:23,835 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:23,835 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:23,835 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:23,835 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:23,835 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:23,835 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:23,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:56:23,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:56:23,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:23,836 DEBUG [Thread-4110] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:56:23,836 DEBUG [Thread-4110] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:56:23,837 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:56:23,936 DEBUG [Thread-4110] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:56:23,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:56:23,939 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:56:23,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:56:23,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:56:23,940 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:56:23,940 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:23,940 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:56:23,940 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:56:23,940 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:23,941 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:23,942 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:23,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:23,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:23,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:23,942 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:23,942 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:23,942 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:56:23,942 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:23,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:56:23,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:23,943 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:56:23,943 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:56:23,943 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:56:23,943 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:56:23,943 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:23,943 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:56:23,944 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:56:23,944 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:56:23,944 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:23,944 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. started... 2023-04-12 15:56:23,944 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4870003209f366bc50ef2084e6e2cf03 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-12 15:56:23,944 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=A 2023-04-12 15:56:23,945 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:23,945 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=B 2023-04-12 15:56:23,945 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:23,945 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=C 2023-04-12 15:56:23,945 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:24,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on 4870003209f366bc50ef2084e6e2cf03 2023-04-12 15:56:24,047 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. as already flushing 2023-04-12 15:56:24,065 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:24,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 426 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53392 deadline: 1681315044063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:24,066 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:24,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 435 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53324 deadline: 1681315044064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:24,067 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:24,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 422 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53360 deadline: 1681315044065, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:24,168 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:24,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 428 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53392 deadline: 1681315044166, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:24,169 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:24,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 437 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53324 deadline: 1681315044167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:24,169 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:24,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 424 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53360 deadline: 1681315044168, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:24,357 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230412ba60dbc29813438c8002eaabf1ad524e to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412ba60dbc29813438c8002eaabf1ad524e 2023-04-12 15:56:24,371 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:24,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 430 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53392 deadline: 1681315044369, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:24,374 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:24,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 439 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53324 deadline: 1681315044371, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:24,374 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:24,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 426 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53360 deadline: 1681315044371, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:24,675 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:24,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 432 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53392 deadline: 1681315044672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:24,679 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:24,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 428 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53360 deadline: 1681315044676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:24,679 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:24,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 441 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53324 deadline: 1681315044677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:24,762 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1158, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/A/55cb2a0647d2491984eed76d4ae1d8d4 2023-04-12 15:56:24,771 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=1158 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/B/00fedc1ec4bf48029bdc2b02b8171fd9 2023-04-12 15:56:24,779 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=1158 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/C/cd1b04703da1480aa5511a963382e75d 2023-04-12 15:56:24,783 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/A/55cb2a0647d2491984eed76d4ae1d8d4 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/55cb2a0647d2491984eed76d4ae1d8d4 2023-04-12 15:56:24,786 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/55cb2a0647d2491984eed76d4ae1d8d4, entries=150, sequenceid=1158, filesize=25.4 K 2023-04-12 15:56:24,787 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/B/00fedc1ec4bf48029bdc2b02b8171fd9 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/00fedc1ec4bf48029bdc2b02b8171fd9 2023-04-12 15:56:24,796 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/00fedc1ec4bf48029bdc2b02b8171fd9, entries=150, sequenceid=1158, filesize=11.9 K 2023-04-12 15:56:24,798 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/C/cd1b04703da1480aa5511a963382e75d as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/cd1b04703da1480aa5511a963382e75d 2023-04-12 15:56:24,801 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/cd1b04703da1480aa5511a963382e75d, entries=150, sequenceid=1158, filesize=11.9 K 2023-04-12 15:56:24,801 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 4870003209f366bc50ef2084e6e2cf03 in 857ms, sequenceid=1158, compaction requested=true 2023-04-12 15:56:24,802 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4870003209f366bc50ef2084e6e2cf03: 2023-04-12 15:56:24,802 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:24,802 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:56:24,802 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:56:24,802 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:24,802 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:56:24,802 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:56:24,803 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:24,803 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:24,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:24,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:24,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:24,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:24,804 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:24,804 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:56:24,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:24,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:24,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:24,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:24,805 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:56:24,805 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b4c4028[Count = 0] remaining members to acquire global barrier 2023-04-12 15:56:24,805 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:56:24,805 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:24,807 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:24,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:24,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:24,807 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:24,807 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:56:24,807 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:56:24,807 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:56:24,807 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:56:24,812 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:24,812 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:56:24,812 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:24,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:24,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:24,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:24,813 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:56:24,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:24,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:24,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:24,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:24,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:24,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:24,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:24,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:56:24,815 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:56:24,815 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:56:24,815 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:56:24,815 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:56:24,815 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:56:24,815 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:56:24,817 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:24,817 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:24,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:24,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:24,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:24,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:24,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:24,817 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:24,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:24,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:24,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:24,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:24,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:24,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:24,818 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:24,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:24,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:24,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:24,820 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:24,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:24,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:24,821 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:24,821 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:24,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:24,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:24,822 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:24,822 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:24,822 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:24,822 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:24,822 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:24,822 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:24,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:24,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:24,822 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:24,822 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:56:24,822 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:56:24,822 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:24,822 DEBUG [Thread-4110] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:56:24,822 DEBUG [Thread-4110] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:56:24,922 DEBUG [Thread-4110] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:56:24,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:56:24,924 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:56:24,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:56:24,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:56:24,926 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:56:24,926 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:24,926 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:56:24,926 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:56:24,926 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:24,927 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:24,928 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:24,928 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:24,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:24,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:24,929 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:24,929 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:56:24,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:24,929 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:24,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:56:24,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:24,930 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:56:24,930 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:56:24,930 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:56:24,930 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:56:24,930 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:24,930 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:56:24,930 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:56:24,930 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:56:24,930 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:24,930 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. started... 2023-04-12 15:56:24,930 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4870003209f366bc50ef2084e6e2cf03 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-12 15:56:24,931 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=A 2023-04-12 15:56:24,931 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:24,931 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=B 2023-04-12 15:56:24,931 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:24,931 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=C 2023-04-12 15:56:24,931 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:25,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on 4870003209f366bc50ef2084e6e2cf03 2023-04-12 15:56:25,181 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. as already flushing 2023-04-12 15:56:25,192 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:25,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 438 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53392 deadline: 1681315045190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:25,195 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:25,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 433 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53360 deadline: 1681315045192, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:25,195 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:25,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 446 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53324 deadline: 1681315045192, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:25,295 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:25,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 440 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53392 deadline: 1681315045293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:25,299 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:25,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 435 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53360 deadline: 1681315045296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:25,299 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:25,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 448 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53324 deadline: 1681315045296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:25,342 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041276435b4e8dd248b4b0a8116a0f1ab1d1 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041276435b4e8dd248b4b0a8116a0f1ab1d1 2023-04-12 15:56:25,346 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1178, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/A/25195452765548deb73288a8e93ec5a5 2023-04-12 15:56:25,498 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:25,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 442 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53392 deadline: 1681315045496, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:25,501 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:25,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 437 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53360 deadline: 1681315045500, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:25,503 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:25,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 450 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53324 deadline: 1681315045501, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:25,755 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=1178 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/B/d2cfeaa3102746e895b06bc88df73d5a 2023-04-12 15:56:25,800 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:25,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 444 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53392 deadline: 1681315045799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:25,805 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:25,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 439 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53360 deadline: 1681315045803, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:25,807 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:25,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 452 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53324 deadline: 1681315045806, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,165 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=1178 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/C/ccac69043df34da584a00adbdafc3f9d 2023-04-12 15:56:26,169 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/A/25195452765548deb73288a8e93ec5a5 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/25195452765548deb73288a8e93ec5a5 2023-04-12 15:56:26,173 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/25195452765548deb73288a8e93ec5a5, entries=150, sequenceid=1178, filesize=25.4 K 2023-04-12 15:56:26,173 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/B/d2cfeaa3102746e895b06bc88df73d5a as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/d2cfeaa3102746e895b06bc88df73d5a 2023-04-12 15:56:26,177 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/d2cfeaa3102746e895b06bc88df73d5a, entries=150, sequenceid=1178, filesize=11.9 K 2023-04-12 15:56:26,177 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/C/ccac69043df34da584a00adbdafc3f9d as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/ccac69043df34da584a00adbdafc3f9d 2023-04-12 15:56:26,180 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/ccac69043df34da584a00adbdafc3f9d, entries=150, sequenceid=1178, filesize=11.9 K 2023-04-12 15:56:26,181 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=67.09 KB/68700 for 4870003209f366bc50ef2084e6e2cf03 in 1251ms, sequenceid=1178, compaction requested=true 2023-04-12 15:56:26,181 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4870003209f366bc50ef2084e6e2cf03: 2023-04-12 15:56:26,181 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:26,181 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:56:26,181 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:56:26,181 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,181 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:56:26,181 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:56:26,182 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:26,182 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:26,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:26,183 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:26,183 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:56:26,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:26,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:26,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:26,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:26,184 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:56:26,184 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@450f6fa[Count = 0] remaining members to acquire global barrier 2023-04-12 15:56:26,184 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:56:26,184 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:26,185 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:26,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:26,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:26,186 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,186 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:56:26,186 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:56:26,186 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:56:26,186 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:56:26,187 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,187 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:56:26,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:26,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:26,187 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:26,187 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:56:26,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:26,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:26,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:26,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:26,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:26,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:56:26,190 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:56:26,190 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:56:26,190 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:56:26,190 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:56:26,190 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:56:26,190 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:56:26,192 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:26,192 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:26,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:26,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:26,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:26,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:26,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:26,192 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:26,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:26,193 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:26,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:26,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:26,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:26,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:26,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:26,194 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:26,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:26,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,196 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,196 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:26,196 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:26,196 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,196 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:26,196 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:26,196 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:56:26,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:26,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:26,196 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:26,196 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:26,196 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:26,196 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:56:26,196 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:26,196 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:56:26,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:26,197 DEBUG [Thread-4110] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:56:26,197 DEBUG [Thread-4110] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:56:26,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:26,297 DEBUG [Thread-4110] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:56:26,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:56:26,299 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:56:26,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:56:26,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:56:26,300 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:56:26,301 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:26,301 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:56:26,301 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:56:26,301 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:26,301 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:26,302 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,302 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:26,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:26,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:26,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:26,303 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,303 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:56:26,303 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:26,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:56:26,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:26,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on 4870003209f366bc50ef2084e6e2cf03 2023-04-12 15:56:26,304 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4870003209f366bc50ef2084e6e2cf03 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-12 15:56:26,304 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=A 2023-04-12 15:56:26,303 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:56:26,304 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:26,304 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=B 2023-04-12 15:56:26,304 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:26,304 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=C 2023-04-12 15:56:26,304 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:56:26,304 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:56:26,304 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:26,304 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:56:26,304 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:26,304 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:56:26,305 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:56:26,305 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:56:26,305 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:26,305 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. started... 2023-04-12 15:56:26,305 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. as already flushing 2023-04-12 15:56:26,305 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:26,305 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:56:26,305 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:56:26,305 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,305 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:56:26,305 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:56:26,307 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,307 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:26,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:26,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:26,307 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:26,307 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:56:26,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:26,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:26,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:26,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:26,310 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:56:26,310 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:56:26,310 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:26,310 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c98e772[Count = 0] remaining members to acquire global barrier 2023-04-12 15:56:26,311 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:26,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:26,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:26,312 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:56:26,312 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:56:26,312 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:56:26,312 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,312 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:56:26,314 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,314 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:56:26,314 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:26,314 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:56:26,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:26,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:26,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:26,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:26,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:26,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:26,316 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041271256f9021ee4af6843ca689e34ae788 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041271256f9021ee4af6843ca689e34ae788 2023-04-12 15:56:26,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:26,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:56:26,317 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:56:26,317 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:56:26,317 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:56:26,318 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:56:26,318 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:56:26,318 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:56:26,319 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:26,319 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:26,320 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:26,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:26,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:26,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:26,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:26,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:26,320 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:26,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:26,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:26,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:26,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:26,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:26,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:26,322 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:26,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:26,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,324 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,324 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:26,324 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:26,324 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:26,324 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,325 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:26,325 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:26,325 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:26,324 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:26,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:26,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:26,324 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:56:26,325 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:56:26,324 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:26,325 DEBUG [Thread-4110] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:56:26,325 DEBUG [Thread-4110] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:56:26,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:26,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:26,326 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:56:26,329 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:26,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 458 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53324 deadline: 1681315046326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,330 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:26,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 453 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53392 deadline: 1681315046327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,330 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:26,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 446 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53360 deadline: 1681315046327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,425 DEBUG [Thread-4110] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:56:26,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:56:26,427 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:56:26,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:56:26,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:56:26,429 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:56:26,429 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:26,429 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:56:26,429 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:56:26,430 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:26,430 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:26,431 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:26,431 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:26,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:26,431 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:26,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 460 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53324 deadline: 1681315046430, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,432 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,432 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:56:26,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:26,432 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:26,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:56:26,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:26,433 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:56:26,433 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:56:26,433 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:56:26,433 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:56:26,433 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:26,433 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:56:26,433 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:26,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 455 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53392 deadline: 1681315046431, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,433 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:56:26,433 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:56:26,434 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:26,433 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:26,434 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. started... 2023-04-12 15:56:26,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 448 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53360 deadline: 1681315046431, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,434 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. as already flushing 2023-04-12 15:56:26,434 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:26,434 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:56:26,434 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:56:26,434 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,434 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:56:26,434 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:56:26,439 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,439 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:26,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:26,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:26,439 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:26,439 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:56:26,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:26,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:26,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:26,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:26,441 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:56:26,441 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47690118[Count = 0] remaining members to acquire global barrier 2023-04-12 15:56:26,441 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:56:26,442 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:26,443 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:26,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:26,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:26,443 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:56:26,443 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:56:26,443 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,443 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:56:26,443 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:56:26,444 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:56:26,444 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:26,444 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,444 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:56:26,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:26,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:26,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:26,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:26,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:26,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:26,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:26,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:56:26,447 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:56:26,447 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:56:26,447 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:56:26,447 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:56:26,447 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:56:26,447 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:56:26,448 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:26,448 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:26,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:26,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:26,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:26,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:26,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:26,448 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:26,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:26,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:26,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:26,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:26,449 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:26,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:26,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:26,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,451 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:26,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:26,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,456 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,456 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:26,456 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:26,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:26,456 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:56:26,456 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:26,456 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:56:26,456 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:56:26,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:26,456 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:26,456 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,456 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:26,456 DEBUG [Thread-4110] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:56:26,456 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:26,456 DEBUG [Thread-4110] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:56:26,456 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:26,457 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:26,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:26,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:26,557 DEBUG [Thread-4110] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:56:26,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:56:26,559 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:56:26,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:56:26,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:56:26,560 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:56:26,560 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:26,560 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:56:26,560 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:56:26,561 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:26,561 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:26,562 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,562 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:26,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:26,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:26,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:26,563 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,563 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:56:26,563 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:26,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:56:26,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:26,564 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:56:26,564 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:56:26,564 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:56:26,564 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:56:26,564 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:26,564 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:56:26,564 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:56:26,564 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:56:26,564 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:26,564 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. started... 2023-04-12 15:56:26,565 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. as already flushing 2023-04-12 15:56:26,565 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:26,565 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:56:26,565 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:56:26,565 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,565 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:56:26,565 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:56:26,566 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,566 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:26,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:26,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:26,567 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:26,567 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:56:26,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:26,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:26,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:26,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:26,568 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:56:26,568 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e1d54dd[Count = 0] remaining members to acquire global barrier 2023-04-12 15:56:26,568 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:56:26,568 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:26,571 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:26,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:26,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:26,571 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:56:26,571 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:56:26,571 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:56:26,571 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,571 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:56:26,572 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,572 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:56:26,572 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:26,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:26,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:26,572 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:56:26,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:26,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:26,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:26,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:26,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:26,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:56:26,575 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:56:26,575 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:56:26,575 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:56:26,575 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:56:26,575 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:56:26,575 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:56:26,577 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:26,577 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:26,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:26,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:26,577 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:26,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:26,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:26,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:26,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:26,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:26,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:26,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:26,578 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:26,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:26,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:26,580 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:26,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:26,582 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,582 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:26,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:56:26,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:56:26,582 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:26,582 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:26,583 DEBUG [Thread-4110] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:56:26,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:26,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:26,582 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:26,583 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,583 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:26,583 DEBUG [Thread-4110] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:56:26,583 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:26,583 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:56:26,583 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:26,584 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:26,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:26,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:26,637 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:26,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 462 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53324 deadline: 1681315046633, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,637 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:26,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 450 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53360 deadline: 1681315046634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,637 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:26,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 457 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53392 deadline: 1681315046635, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,683 DEBUG [Thread-4110] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:56:26,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:56:26,685 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:56:26,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:56:26,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:56:26,687 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:56:26,687 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:26,687 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:56:26,687 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:56:26,687 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:26,687 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:26,689 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,689 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:26,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:26,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:26,690 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,690 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:56:26,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:26,690 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:26,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:56:26,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:26,690 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:56:26,690 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:56:26,691 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:56:26,691 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:56:26,691 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:26,691 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:56:26,691 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:56:26,691 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:56:26,691 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:26,691 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. started... 2023-04-12 15:56:26,691 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. as already flushing 2023-04-12 15:56:26,692 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:26,692 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:56:26,692 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:56:26,692 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,692 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:56:26,692 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:56:26,693 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:26,693 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:26,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:26,693 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:26,694 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:56:26,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:26,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:26,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:26,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:26,695 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:56:26,695 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d798ac2[Count = 0] remaining members to acquire global barrier 2023-04-12 15:56:26,695 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:56:26,695 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:26,696 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:26,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:26,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:26,696 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:56:26,696 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:56:26,696 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:56:26,696 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,697 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:56:26,698 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:26,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:26,698 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:56:26,698 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:26,698 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:56:26,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:26,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:26,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:26,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:26,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:26,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:56:26,700 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:56:26,700 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:56:26,700 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:56:26,700 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:56:26,700 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:56:26,700 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:56:26,703 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:26,703 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:26,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:26,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:26,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:26,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:26,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:26,703 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:26,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:26,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:26,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:26,704 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:26,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:26,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:26,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:26,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,705 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:26,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:26,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,707 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:26,707 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:26,707 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:26,707 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:26,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:26,707 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:56:26,707 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:56:26,707 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:26,708 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:26,708 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,708 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:26,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:26,708 DEBUG [Thread-4110] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:56:26,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:26,708 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:26,708 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:26,708 DEBUG [Thread-4110] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:56:26,721 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1192, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/A/1b13c1ead2d843fbac2fffba75172f78 2023-04-12 15:56:26,731 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=1192 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/B/a082e532720e44a6926ed542fc275287 2023-04-12 15:56:26,808 DEBUG [Thread-4110] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:56:26,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:56:26,810 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:56:26,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:56:26,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:56:26,812 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:56:26,812 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:26,812 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:56:26,812 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:56:26,813 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:26,813 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:26,814 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:26,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:26,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:26,814 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,814 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,814 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:56:26,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:26,815 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:26,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:56:26,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:26,815 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:56:26,815 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:56:26,815 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:56:26,815 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:56:26,816 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:26,816 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:56:26,816 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:56:26,816 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:56:26,816 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:26,816 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. started... 2023-04-12 15:56:26,816 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. as already flushing 2023-04-12 15:56:26,816 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:26,816 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:56:26,816 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:56:26,816 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,816 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:56:26,816 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:56:26,818 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:26,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:26,818 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:26,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:26,818 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:26,818 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:56:26,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:26,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:26,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:26,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:56:26,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66415168[Count = 0] remaining members to acquire global barrier 2023-04-12 15:56:26,820 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:56:26,820 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:26,822 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:26,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:26,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:26,822 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:56:26,822 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,822 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:56:26,822 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:56:26,822 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:56:26,823 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:26,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:26,824 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:56:26,824 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:26,824 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:56:26,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:26,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:26,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:26,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:26,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:26,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:56:26,826 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:56:26,826 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:56:26,827 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:56:26,827 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:56:26,827 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:56:26,827 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:56:26,828 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:26,828 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:26,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:26,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:26,828 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:26,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:26,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:26,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:26,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:26,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:26,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:26,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:26,829 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:26,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:26,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:26,830 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:26,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:26,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,836 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:26,836 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:26,836 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,836 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:26,836 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:26,836 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,836 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:26,836 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:56:26,836 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:26,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:56:26,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:56:26,836 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:26,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:26,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:26,837 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:26,837 DEBUG [Thread-4110] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:56:26,837 DEBUG [Thread-4110] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:56:26,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:26,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:26,937 DEBUG [Thread-4110] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:56:26,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:56:26,939 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:56:26,940 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:26,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 464 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53324 deadline: 1681315046938, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,940 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:26,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 452 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53360 deadline: 1681315046938, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:56:26,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:56:26,940 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:56:26,940 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:26,941 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:56:26,941 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:56:26,941 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:26,941 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:26,941 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:26,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 459 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53392 deadline: 1681315046939, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,943 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:26,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:26,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:26,943 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:26,944 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,944 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:56:26,944 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:26,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:56:26,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:26,944 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:56:26,945 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:56:26,945 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:56:26,945 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:56:26,945 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:26,945 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:56:26,945 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:56:26,945 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:56:26,945 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:26,945 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. started... 2023-04-12 15:56:26,945 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. as already flushing 2023-04-12 15:56:26,945 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:26,945 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:56:26,945 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:56:26,945 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,945 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:56:26,945 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:56:26,947 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,947 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:26,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:26,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:26,947 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:26,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:26,947 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:56:26,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:26,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:26,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:26,948 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:56:26,948 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69afed79[Count = 0] remaining members to acquire global barrier 2023-04-12 15:56:26,948 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:56:26,949 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:26,950 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:26,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:26,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:26,950 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:56:26,950 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:56:26,950 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:56:26,950 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,950 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:56:26,951 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,951 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:56:26,951 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:26,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:26,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:26,951 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:56:26,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:26,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:26,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:26,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:26,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:26,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:56:26,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:56:26,954 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:56:26,954 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:56:26,954 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:56:26,954 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:56:26,954 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:56:26,956 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:26,956 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:26,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:26,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:26,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:26,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:26,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:26,956 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:26,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:26,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:26,957 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:26,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:26,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:26,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:26,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:26,958 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:26,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:26,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,960 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:26,960 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:26,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:26,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:26,960 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:56:26,960 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,960 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:56:26,960 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:56:26,960 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:26,960 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:26,961 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:26,961 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:26,961 DEBUG [Thread-4110] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:56:26,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:26,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:26,961 DEBUG [Thread-4110] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:56:26,961 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:26,961 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:26,961 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:27,061 DEBUG [Thread-4110] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:56:27,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:56:27,063 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:56:27,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:56:27,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:56:27,065 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:56:27,065 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:27,065 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:56:27,065 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:56:27,065 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:27,065 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:27,066 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:27,066 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:27,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:27,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:27,067 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:27,067 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:56:27,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:27,067 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:27,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:56:27,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:27,068 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:56:27,068 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 15:56:27,068 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:56:27,068 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:56:27,068 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:27,068 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:56:27,068 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:56:27,068 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:56:27,068 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:27,068 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. started... 2023-04-12 15:56:27,069 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. as already flushing 2023-04-12 15:56:27,069 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:27,069 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:56:27,069 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:56:27,069 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:27,069 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:56:27,069 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:56:27,070 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:27,070 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:27,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:27,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:27,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:27,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:27,071 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:27,071 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:56:27,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:27,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:27,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:27,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:27,073 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:56:27,073 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d00e72d[Count = 0] remaining members to acquire global barrier 2023-04-12 15:56:27,073 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:56:27,073 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:27,080 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:27,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:27,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:27,080 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:56:27,080 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:56:27,080 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:56:27,080 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:27,080 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:56:27,081 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:27,081 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:56:27,081 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:27,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:27,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:27,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:27,082 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:56:27,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:27,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:27,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:27,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:27,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:27,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:27,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:27,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:56:27,084 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:56:27,084 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:56:27,084 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:56:27,084 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:56:27,084 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:56:27,084 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:56:27,086 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:27,086 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:27,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:27,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:27,086 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:27,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:27,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:27,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:27,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:27,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:27,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:27,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:27,087 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:27,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:27,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:27,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:27,088 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:27,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:27,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:27,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:27,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:27,090 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:27,090 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:27,090 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:27,090 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:27,090 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:27,090 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:27,090 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:27,090 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:27,090 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:56:27,090 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:56:27,090 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:27,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:27,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:27,090 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:27,091 DEBUG [Thread-4110] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:56:27,091 DEBUG [Thread-4110] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:56:27,091 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 15:56:27,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:27,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:27,140 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=1192 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/C/d778e6f30ec744c3ae2fa9ec58771358 2023-04-12 15:56:27,153 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/A/1b13c1ead2d843fbac2fffba75172f78 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/1b13c1ead2d843fbac2fffba75172f78 2023-04-12 15:56:27,156 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/1b13c1ead2d843fbac2fffba75172f78, entries=200, sequenceid=1192, filesize=32.3 K 2023-04-12 15:56:27,157 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/B/a082e532720e44a6926ed542fc275287 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/a082e532720e44a6926ed542fc275287 2023-04-12 15:56:27,160 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/a082e532720e44a6926ed542fc275287, entries=150, sequenceid=1192, filesize=11.9 K 2023-04-12 15:56:27,160 DEBUG [MemStoreFlusher.0] regionserver.StoreScanner(961): StoreScanner already closing. There is no need to updateReaders 2023-04-12 15:56:27,161 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/C/d778e6f30ec744c3ae2fa9ec58771358 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/d778e6f30ec744c3ae2fa9ec58771358 2023-04-12 15:56:27,164 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/d778e6f30ec744c3ae2fa9ec58771358, entries=150, sequenceid=1192, filesize=11.9 K 2023-04-12 15:56:27,164 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 4870003209f366bc50ef2084e6e2cf03 in 861ms, sequenceid=1192, compaction requested=true 2023-04-12 15:56:27,164 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4870003209f366bc50ef2084e6e2cf03: 2023-04-12 15:56:27,165 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:56:27,165 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:56:27,165 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-12 15:56:27,165 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:56:27,165 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-12 15:56:27,166 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 138530 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-12 15:56:27,166 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1912): 4870003209f366bc50ef2084e6e2cf03/A is initiating minor compaction (all files) 2023-04-12 15:56:27,166 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 62224 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-12 15:56:27,166 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4870003209f366bc50ef2084e6e2cf03/A in TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:27,166 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1912): 4870003209f366bc50ef2084e6e2cf03/B is initiating minor compaction (all files) 2023-04-12 15:56:27,166 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/11f9a24bd9ea4268881aaf2d5e046a6b, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/211b68e577744284a3580da1d4f6ce35, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/55cb2a0647d2491984eed76d4ae1d8d4, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/25195452765548deb73288a8e93ec5a5, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/1b13c1ead2d843fbac2fffba75172f78] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp, totalSize=135.3 K 2023-04-12 15:56:27,166 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4870003209f366bc50ef2084e6e2cf03/B in TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:27,167 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/139a72300c38492c9fb4159827dea1c6, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/d429a62d40b34216ba046da40ebffff0, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/00fedc1ec4bf48029bdc2b02b8171fd9, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/d2cfeaa3102746e895b06bc88df73d5a, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/a082e532720e44a6926ed542fc275287] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp, totalSize=60.8 K 2023-04-12 15:56:27,167 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 139a72300c38492c9fb4159827dea1c6, keycount=150, bloomtype=ROW, size=13.2 K, encoding=NONE, compression=NONE, seqNum=1108, earliestPutTs=1681314982707 2023-04-12 15:56:27,167 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 11f9a24bd9ea4268881aaf2d5e046a6b, keycount=150, bloomtype=ROW, size=26.7 K, encoding=NONE, compression=NONE, seqNum=1108, earliestPutTs=1681314982707 2023-04-12 15:56:27,167 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting d429a62d40b34216ba046da40ebffff0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1146, earliestPutTs=1681314982809 2023-04-12 15:56:27,168 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 211b68e577744284a3580da1d4f6ce35, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=1146, earliestPutTs=1681314982809 2023-04-12 15:56:27,168 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 00fedc1ec4bf48029bdc2b02b8171fd9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1158, earliestPutTs=1681314983428 2023-04-12 15:56:27,168 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 55cb2a0647d2491984eed76d4ae1d8d4, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=1158, earliestPutTs=1681314983428 2023-04-12 15:56:27,168 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting d2cfeaa3102746e895b06bc88df73d5a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1178, earliestPutTs=1681314984059 2023-04-12 15:56:27,168 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 25195452765548deb73288a8e93ec5a5, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=1178, earliestPutTs=1681314984059 2023-04-12 15:56:27,168 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting a082e532720e44a6926ed542fc275287, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1192, earliestPutTs=1681314985189 2023-04-12 15:56:27,169 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 1b13c1ead2d843fbac2fffba75172f78, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=1192, earliestPutTs=1681314985189 2023-04-12 15:56:27,178 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] throttle.PressureAwareThroughputController(145): 4870003209f366bc50ef2084e6e2cf03#B#compaction#680 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:56:27,186 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4870003209f366bc50ef2084e6e2cf03#A#compaction#681 average throughput is 1.97 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:56:27,191 DEBUG [Thread-4110] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:56:27,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:56:27,193 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 15:56:27,199 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/B/2e4048381027401e8a9726c6bce77086 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/2e4048381027401e8a9726c6bce77086 2023-04-12 15:56:27,203 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 4870003209f366bc50ef2084e6e2cf03/B of 4870003209f366bc50ef2084e6e2cf03 into 2e4048381027401e8a9726c6bce77086(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:56:27,203 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4870003209f366bc50ef2084e6e2cf03: 2023-04-12 15:56:27,203 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03., storeName=4870003209f366bc50ef2084e6e2cf03/B, priority=11, startTime=1681314987165; duration=0sec 2023-04-12 15:56:27,203 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:56:27,203 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-12 15:56:27,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 15:56:27,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 15:56:27,204 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 15:56:27,204 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:27,204 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 15:56:27,204 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 15:56:27,204 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:27,204 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:27,205 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 62190 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-12 15:56:27,205 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1912): 4870003209f366bc50ef2084e6e2cf03/C is initiating minor compaction (all files) 2023-04-12 15:56:27,205 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4870003209f366bc50ef2084e6e2cf03/C in TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:27,205 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/81a3c88580514d57a41154eaa484c309, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/5a07130d038d402e8777cf9df6fe0a8f, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/cd1b04703da1480aa5511a963382e75d, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/ccac69043df34da584a00adbdafc3f9d, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/d778e6f30ec744c3ae2fa9ec58771358] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp, totalSize=60.7 K 2023-04-12 15:56:27,205 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 81a3c88580514d57a41154eaa484c309, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=1108, earliestPutTs=1681314982707 2023-04-12 15:56:27,205 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 5a07130d038d402e8777cf9df6fe0a8f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1146, earliestPutTs=1681314982809 2023-04-12 15:56:27,206 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting cd1b04703da1480aa5511a963382e75d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1158, earliestPutTs=1681314983428 2023-04-12 15:56:27,206 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:27,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:27,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:27,206 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:27,206 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting ccac69043df34da584a00adbdafc3f9d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1178, earliestPutTs=1681314984059 2023-04-12 15:56:27,206 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:27,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:27,206 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 15:56:27,206 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting d778e6f30ec744c3ae2fa9ec58771358, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1192, earliestPutTs=1681314985189 2023-04-12 15:56:27,207 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:27,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 15:56:27,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:27,207 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 15:56:27,207 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 15:56:27,207 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 15:56:27,207 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 15:56:27,207 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 15:56:27,208 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 15:56:27,208 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 15:56:27,208 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:27,208 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. started... 2023-04-12 15:56:27,208 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4870003209f366bc50ef2084e6e2cf03 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-12 15:56:27,208 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=A 2023-04-12 15:56:27,208 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:27,208 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=B 2023-04-12 15:56:27,208 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:27,208 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=C 2023-04-12 15:56:27,209 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:27,222 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] throttle.PressureAwareThroughputController(145): 4870003209f366bc50ef2084e6e2cf03#C#compaction#682 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:56:27,265 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/C/9d2ea6db7209483cb4030d2d88de66b2 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/9d2ea6db7209483cb4030d2d88de66b2 2023-04-12 15:56:27,270 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 4870003209f366bc50ef2084e6e2cf03/C of 4870003209f366bc50ef2084e6e2cf03 into 9d2ea6db7209483cb4030d2d88de66b2(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:56:27,270 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4870003209f366bc50ef2084e6e2cf03: 2023-04-12 15:56:27,270 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03., storeName=4870003209f366bc50ef2084e6e2cf03/C, priority=11, startTime=1681314987165; duration=0sec 2023-04-12 15:56:27,270 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:56:27,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on 4870003209f366bc50ef2084e6e2cf03 2023-04-12 15:56:27,443 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. as already flushing 2023-04-12 15:56:27,458 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:27,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 469 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53324 deadline: 1681315047454, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:27,462 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:27,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 464 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53392 deadline: 1681315047459, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:27,462 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:27,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38905] ipc.CallRunner(144): callId: 459 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53360 deadline: 1681315047459, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:27,466 DEBUG [Thread-4113] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x32bf16cd to 127.0.0.1:60154 2023-04-12 15:56:27,466 DEBUG [Thread-4113] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 15:56:27,466 DEBUG [Thread-4111] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3d765193 to 127.0.0.1:60154 2023-04-12 15:56:27,466 DEBUG [Thread-4111] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 15:56:27,466 DEBUG [Thread-4115] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x28164190 to 127.0.0.1:60154 2023-04-12 15:56:27,467 DEBUG [Thread-4115] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 15:56:27,470 DEBUG [Thread-4119] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x01453d93 to 127.0.0.1:60154 2023-04-12 15:56:27,470 DEBUG [Thread-4119] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 15:56:27,470 DEBUG [Thread-4117] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x308b3f39 to 127.0.0.1:60154 2023-04-12 15:56:27,470 DEBUG [Thread-4117] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 15:56:27,560 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:27,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 471 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53324 deadline: 1681315047560, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:27,563 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:27,563 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:27,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 466 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53392 deadline: 1681315047563, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:27,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 461 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53360 deadline: 1681315047563, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:27,664 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230412ac9d7b1fb45a46b3a5e8be1276788c1f to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412ac9d7b1fb45a46b3a5e8be1276788c1f 2023-04-12 15:56:27,761 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:27,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 473 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53324 deadline: 1681315047761, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:27,764 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:27,764 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:27,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38905] ipc.CallRunner(144): callId: 468 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53392 deadline: 1681315047764, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:27,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 463 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53360 deadline: 1681315047764, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:28,018 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/A/3f76e3d7466a46d2b294c1a705583f3f as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/3f76e3d7466a46d2b294c1a705583f3f 2023-04-12 15:56:28,022 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 4870003209f366bc50ef2084e6e2cf03/A of 4870003209f366bc50ef2084e6e2cf03 into 3f76e3d7466a46d2b294c1a705583f3f(size=26.8 K), total size for store is 26.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:56:28,022 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4870003209f366bc50ef2084e6e2cf03: 2023-04-12 15:56:28,022 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03., storeName=4870003209f366bc50ef2084e6e2cf03/A, priority=11, startTime=1681314987164; duration=0sec 2023-04-12 15:56:28,022 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:56:28,064 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:28,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 475 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53324 deadline: 1681315048064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:28,066 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:28,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 470 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53392 deadline: 1681315048066, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:28,067 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 15:56:28,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] ipc.CallRunner(144): callId: 465 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53360 deadline: 1681315048067, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:28,070 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1211, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/A/5a3cb07824f14ef2b802a9072d6db8ac 2023-04-12 15:56:28,077 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=1211 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/B/18c974752f8f4157a7c01ff25d3586ec 2023-04-12 15:56:28,084 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=1211 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/C/804b2e48cf6e483baa6276c67d076986 2023-04-12 15:56:28,087 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/A/5a3cb07824f14ef2b802a9072d6db8ac as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/5a3cb07824f14ef2b802a9072d6db8ac 2023-04-12 15:56:28,089 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/5a3cb07824f14ef2b802a9072d6db8ac, entries=150, sequenceid=1211, filesize=25.4 K 2023-04-12 15:56:28,090 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/B/18c974752f8f4157a7c01ff25d3586ec as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/18c974752f8f4157a7c01ff25d3586ec 2023-04-12 15:56:28,092 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/18c974752f8f4157a7c01ff25d3586ec, entries=150, sequenceid=1211, filesize=11.9 K 2023-04-12 15:56:28,093 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/C/804b2e48cf6e483baa6276c67d076986 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/804b2e48cf6e483baa6276c67d076986 2023-04-12 15:56:28,096 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/804b2e48cf6e483baa6276c67d076986, entries=150, sequenceid=1211, filesize=11.9 K 2023-04-12 15:56:28,096 INFO [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 4870003209f366bc50ef2084e6e2cf03 in 888ms, sequenceid=1211, compaction requested=false 2023-04-12 15:56:28,096 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4870003209f366bc50ef2084e6e2cf03: 2023-04-12 15:56:28,096 DEBUG [rs(jenkins-hbase4.apache.org,38905,1681314818739)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:28,096 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 15:56:28,096 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 15:56:28,096 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:28,096 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 15:56:28,096 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 15:56:28,098 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:28,098 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:28,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:28,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:28,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:28,098 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:28,098 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 15:56:28,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:28,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:28,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:28,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:28,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:28,099 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38905,1681314818739' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 15:56:28,099 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c9e5dd5[Count = 0] remaining members to acquire global barrier 2023-04-12 15:56:28,099 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 15:56:28,099 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:28,100 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:28,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:28,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:28,100 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 15:56:28,100 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:28,100 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 15:56:28,100 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 15:56:28,100 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38905,1681314818739' in zk 2023-04-12 15:56:28,103 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 15:56:28,103 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:28,103 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:28,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:28,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:28,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:28,103 DEBUG [member: 'jenkins-hbase4.apache.org,38905,1681314818739' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 15:56:28,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:28,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:28,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:28,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:28,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:28,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:28,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:28,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38905,1681314818739': 2023-04-12 15:56:28,105 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38905,1681314818739' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 15:56:28,105 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 15:56:28,105 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 15:56:28,105 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 15:56:28,105 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 15:56:28,105 INFO [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 15:56:28,106 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:28,106 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:28,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:28,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 15:56:28,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 15:56:28,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:28,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:28,106 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:28,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 15:56:28,107 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:28,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:28,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:28,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:28,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:28,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 15:56:28,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:28,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:28,108 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:28,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 15:56:28,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 15:56:28,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:28,111 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:28,111 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 15:56:28,111 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:28,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 15:56:28,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 15:56:28,111 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 15:56:28,112 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 15:56:28,111 DEBUG [(jenkins-hbase4.apache.org,36273,1681314817519)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 15:56:28,111 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 15:56:28,111 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 15:56:28,112 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:28,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 15:56:28,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 15:56:28,112 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:28,112 DEBUG [Thread-4110] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 15:56:28,112 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 15:56:28,112 DEBUG [Thread-4110] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 15:56:28,112 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 15:56:28,212 DEBUG [Thread-4110] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 15:56:28,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 15:56:28,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38905] regionserver.HRegion(9158): Flush requested on 4870003209f366bc50ef2084e6e2cf03 2023-04-12 15:56:28,568 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4870003209f366bc50ef2084e6e2cf03 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-12 15:56:28,568 DEBUG [Thread-4104] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x06799455 to 127.0.0.1:60154 2023-04-12 15:56:28,568 DEBUG [Thread-4102] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x08b57aae to 127.0.0.1:60154 2023-04-12 15:56:28,568 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=A 2023-04-12 15:56:28,568 DEBUG [Thread-4102] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 15:56:28,568 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:28,568 DEBUG [Thread-4104] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 15:56:28,568 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=B 2023-04-12 15:56:28,569 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:28,569 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=C 2023-04-12 15:56:28,569 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:28,573 DEBUG [Thread-4106] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x66c25427 to 127.0.0.1:60154 2023-04-12 15:56:28,573 DEBUG [Thread-4106] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 15:56:28,982 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230412c31bed3ae125489e86fb0cc8325bb214 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412c31bed3ae125489e86fb0cc8325bb214 2023-04-12 15:56:29,386 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1229, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/A/bb149c5e0e5d416ea0cce34b5b956223 2023-04-12 15:56:29,796 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=1229 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/B/1962da4b34644b0999b641f120261a92 2023-04-12 15:56:30,203 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=1229 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/C/b00a6f42dbbf4f7f97df195a8fa04339 2023-04-12 15:56:30,207 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/A/bb149c5e0e5d416ea0cce34b5b956223 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/bb149c5e0e5d416ea0cce34b5b956223 2023-04-12 15:56:30,209 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/bb149c5e0e5d416ea0cce34b5b956223, entries=150, sequenceid=1229, filesize=25.4 K 2023-04-12 15:56:30,210 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/B/1962da4b34644b0999b641f120261a92 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/1962da4b34644b0999b641f120261a92 2023-04-12 15:56:30,213 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/1962da4b34644b0999b641f120261a92, entries=150, sequenceid=1229, filesize=11.9 K 2023-04-12 15:56:30,213 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/C/b00a6f42dbbf4f7f97df195a8fa04339 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/b00a6f42dbbf4f7f97df195a8fa04339 2023-04-12 15:56:30,216 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/b00a6f42dbbf4f7f97df195a8fa04339, entries=150, sequenceid=1229, filesize=11.9 K 2023-04-12 15:56:30,216 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=6.71 KB/6870 for 4870003209f366bc50ef2084e6e2cf03 in 1648ms, sequenceid=1229, compaction requested=true 2023-04-12 15:56:30,217 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4870003209f366bc50ef2084e6e2cf03: 2023-04-12 15:56:30,217 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:56:30,217 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 15:56:30,217 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 15:56:30,217 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:56:30,217 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:56:30,217 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 79568 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 15:56:30,218 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38022 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 15:56:30,218 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1912): 4870003209f366bc50ef2084e6e2cf03/A is initiating minor compaction (all files) 2023-04-12 15:56:30,218 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1912): 4870003209f366bc50ef2084e6e2cf03/B is initiating minor compaction (all files) 2023-04-12 15:56:30,218 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4870003209f366bc50ef2084e6e2cf03/A in TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:30,218 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4870003209f366bc50ef2084e6e2cf03/B in TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:30,218 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/3f76e3d7466a46d2b294c1a705583f3f, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/5a3cb07824f14ef2b802a9072d6db8ac, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/bb149c5e0e5d416ea0cce34b5b956223] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp, totalSize=77.7 K 2023-04-12 15:56:30,218 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/2e4048381027401e8a9726c6bce77086, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/18c974752f8f4157a7c01ff25d3586ec, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/1962da4b34644b0999b641f120261a92] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp, totalSize=37.1 K 2023-04-12 15:56:30,218 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 3f76e3d7466a46d2b294c1a705583f3f, keycount=150, bloomtype=ROW, size=26.8 K, encoding=NONE, compression=NONE, seqNum=1192, earliestPutTs=1681314985189 2023-04-12 15:56:30,218 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 2e4048381027401e8a9726c6bce77086, keycount=150, bloomtype=ROW, size=13.3 K, encoding=NONE, compression=NONE, seqNum=1192, earliestPutTs=1681314985189 2023-04-12 15:56:30,218 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting 5a3cb07824f14ef2b802a9072d6db8ac, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=1211, earliestPutTs=1681314986325 2023-04-12 15:56:30,219 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 18c974752f8f4157a7c01ff25d3586ec, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1211, earliestPutTs=1681314986325 2023-04-12 15:56:30,219 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] compactions.Compactor(207): Compacting bb149c5e0e5d416ea0cce34b5b956223, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=1229, earliestPutTs=1681314987449 2023-04-12 15:56:30,219 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 1962da4b34644b0999b641f120261a92, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1229, earliestPutTs=1681314987449 2023-04-12 15:56:30,225 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] throttle.PressureAwareThroughputController(145): 4870003209f366bc50ef2084e6e2cf03#B#compaction#689 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:56:30,229 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4870003209f366bc50ef2084e6e2cf03#A#compaction#690 average throughput is 6.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:56:30,232 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/B/45a607c53ac14188b1ef9b88df719aed as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/45a607c53ac14188b1ef9b88df719aed 2023-04-12 15:56:30,235 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4870003209f366bc50ef2084e6e2cf03/B of 4870003209f366bc50ef2084e6e2cf03 into 45a607c53ac14188b1ef9b88df719aed(size=13.4 K), total size for store is 13.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:56:30,235 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4870003209f366bc50ef2084e6e2cf03: 2023-04-12 15:56:30,235 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03., storeName=4870003209f366bc50ef2084e6e2cf03/B, priority=13, startTime=1681314990217; duration=0sec 2023-04-12 15:56:30,236 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 15:56:30,236 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 15:56:30,237 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37988 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 15:56:30,237 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1912): 4870003209f366bc50ef2084e6e2cf03/C is initiating minor compaction (all files) 2023-04-12 15:56:30,237 DEBUG [Thread-4100] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4db22237 to 127.0.0.1:60154 2023-04-12 15:56:30,237 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4870003209f366bc50ef2084e6e2cf03/C in TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:30,237 DEBUG [Thread-4100] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 15:56:30,237 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/9d2ea6db7209483cb4030d2d88de66b2, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/804b2e48cf6e483baa6276c67d076986, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/b00a6f42dbbf4f7f97df195a8fa04339] into tmpdir=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp, totalSize=37.1 K 2023-04-12 15:56:30,237 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 9d2ea6db7209483cb4030d2d88de66b2, keycount=150, bloomtype=ROW, size=13.3 K, encoding=NONE, compression=NONE, seqNum=1192, earliestPutTs=1681314985189 2023-04-12 15:56:30,238 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting 804b2e48cf6e483baa6276c67d076986, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1211, earliestPutTs=1681314986325 2023-04-12 15:56:30,238 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] compactions.Compactor(207): Compacting b00a6f42dbbf4f7f97df195a8fa04339, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1229, earliestPutTs=1681314987449 2023-04-12 15:56:30,243 DEBUG [Thread-4108] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4f4eab6d to 127.0.0.1:60154 2023-04-12 15:56:30,243 DEBUG [Thread-4108] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 15:56:30,243 INFO [Listener at localhost/43583] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-12 15:56:30,243 INFO [Listener at localhost/43583] hbase.AcidGuaranteesTestTool(394): wrote 58 2023-04-12 15:56:30,243 INFO [Listener at localhost/43583] hbase.AcidGuaranteesTestTool(394): wrote 325 2023-04-12 15:56:30,243 INFO [Listener at localhost/43583] hbase.AcidGuaranteesTestTool(394): wrote 330 2023-04-12 15:56:30,243 INFO [Listener at localhost/43583] hbase.AcidGuaranteesTestTool(394): wrote 320 2023-04-12 15:56:30,243 INFO [Listener at localhost/43583] hbase.AcidGuaranteesTestTool(394): wrote 59 2023-04-12 15:56:30,243 INFO [Listener at localhost/43583] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-12 15:56:30,243 INFO [Listener at localhost/43583] hbase.AcidGuaranteesTestTool(398): read 4522 2023-04-12 15:56:30,243 INFO [Listener at localhost/43583] hbase.AcidGuaranteesTestTool(398): read 4428 2023-04-12 15:56:30,243 INFO [Listener at localhost/43583] hbase.AcidGuaranteesTestTool(398): read 4370 2023-04-12 15:56:30,243 INFO [Listener at localhost/43583] hbase.AcidGuaranteesTestTool(398): read 4539 2023-04-12 15:56:30,243 INFO [Listener at localhost/43583] hbase.AcidGuaranteesTestTool(398): read 4428 2023-04-12 15:56:30,243 INFO [Listener at localhost/43583] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-12 15:56:30,243 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] throttle.PressureAwareThroughputController(145): 4870003209f366bc50ef2084e6e2cf03#C#compaction#691 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 15:56:30,244 INFO [Listener at localhost/43583] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-12 15:56:30,244 DEBUG [Listener at localhost/43583] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5f9e8654 to 127.0.0.1:60154 2023-04-12 15:56:30,244 DEBUG [Listener at localhost/43583] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 15:56:30,244 INFO [Listener at localhost/43583] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-12 15:56:30,245 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-12 15:56:30,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure2.ProcedureExecutor(1029): Stored pid=62, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-12 15:56:30,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-12 15:56:30,248 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681314990248"}]},"ts":"1681314990248"} 2023-04-12 15:56:30,248 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-12 15:56:30,250 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-12 15:56:30,250 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=4870003209f366bc50ef2084e6e2cf03, UNASSIGN}] 2023-04-12 15:56:30,251 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/C/d1b650c782ee4b47b269a5ec775cd446 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/d1b650c782ee4b47b269a5ec775cd446 2023-04-12 15:56:30,252 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=4870003209f366bc50ef2084e6e2cf03, UNASSIGN 2023-04-12 15:56:30,252 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=4870003209f366bc50ef2084e6e2cf03, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:30,252 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681314990252"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681314990252"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681314990252"}]},"ts":"1681314990252"} 2023-04-12 15:56:30,253 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=64, ppid=63, state=RUNNABLE; CloseRegionProcedure 4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739}] 2023-04-12 15:56:30,255 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4870003209f366bc50ef2084e6e2cf03/C of 4870003209f366bc50ef2084e6e2cf03 into d1b650c782ee4b47b269a5ec775cd446(size=13.4 K), total size for store is 13.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:56:30,255 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4870003209f366bc50ef2084e6e2cf03: 2023-04-12 15:56:30,255 INFO [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03., storeName=4870003209f366bc50ef2084e6e2cf03/C, priority=13, startTime=1681314990217; duration=0sec 2023-04-12 15:56:30,255 DEBUG [RS:0;jenkins-hbase4:38905-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:56:30,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-12 15:56:30,405 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 4870003209f366bc50ef2084e6e2cf03 2023-04-12 15:56:30,405 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 4870003209f366bc50ef2084e6e2cf03, disabling compactions & flushes 2023-04-12 15:56:30,405 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 1 compactions to complete for region TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:30,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-12 15:56:30,644 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/A/f45beb74c52c4f72946210d2591e2400 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/f45beb74c52c4f72946210d2591e2400 2023-04-12 15:56:30,647 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4870003209f366bc50ef2084e6e2cf03/A of 4870003209f366bc50ef2084e6e2cf03 into f45beb74c52c4f72946210d2591e2400(size=26.9 K), total size for store is 26.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 15:56:30,647 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4870003209f366bc50ef2084e6e2cf03: 2023-04-12 15:56:30,647 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:30,647 INFO [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03., storeName=4870003209f366bc50ef2084e6e2cf03/A, priority=13, startTime=1681314990217; duration=0sec 2023-04-12 15:56:30,647 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:30,647 DEBUG [RS:0;jenkins-hbase4:38905-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 15:56:30,647 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. after waiting 0 ms 2023-04-12 15:56:30,647 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:30,647 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 4870003209f366bc50ef2084e6e2cf03 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-12 15:56:30,648 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=A 2023-04-12 15:56:30,648 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:30,648 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=B 2023-04-12 15:56:30,648 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:30,648 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4870003209f366bc50ef2084e6e2cf03, store=C 2023-04-12 15:56:30,648 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 15:56:30,655 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304127f5e5b2ab661493e90bd6dfe1b7d0ccc to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304127f5e5b2ab661493e90bd6dfe1b7d0ccc 2023-04-12 15:56:30,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-12 15:56:31,060 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1238, memsize=6.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/A/7fa3101e18534606bc52c046289413dc 2023-04-12 15:56:31,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-12 15:56:31,468 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=1238 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/B/7831d6052c1d4a1a873d6a9fad2ae4f9 2023-04-12 15:56:31,876 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=1238 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/C/6a25a77417af41919d48e25d7af3d7b4 2023-04-12 15:56:31,879 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/A/7fa3101e18534606bc52c046289413dc as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/7fa3101e18534606bc52c046289413dc 2023-04-12 15:56:31,881 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/7fa3101e18534606bc52c046289413dc, entries=100, sequenceid=1238, filesize=18.6 K 2023-04-12 15:56:31,882 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/B/7831d6052c1d4a1a873d6a9fad2ae4f9 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/7831d6052c1d4a1a873d6a9fad2ae4f9 2023-04-12 15:56:31,884 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/7831d6052c1d4a1a873d6a9fad2ae4f9, entries=100, sequenceid=1238, filesize=9.5 K 2023-04-12 15:56:31,885 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/.tmp/C/6a25a77417af41919d48e25d7af3d7b4 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/6a25a77417af41919d48e25d7af3d7b4 2023-04-12 15:56:31,887 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/6a25a77417af41919d48e25d7af3d7b4, entries=100, sequenceid=1238, filesize=9.5 K 2023-04-12 15:56:31,888 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=0 B/0 for 4870003209f366bc50ef2084e6e2cf03 in 1241ms, sequenceid=1238, compaction requested=false 2023-04-12 15:56:31,889 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/c3592a2ea53c4b34b78fbc83be19ebe2, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/14632b72474342a197ebfcab156ba9b6, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/648406cb77ba4910b5b3ca50c90ceffe, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/58e51f98e0724a8b8acf6d815d7e9d6b, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/6f51d7e5811b4de6977036482e15daea, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/e4d5a161e3b24a94a83578a20f1c850c, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/4c9ec657b3454f169fc44806f6254a70, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/071fc3308c3b41daa6b97b0e277421f0, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/6d5d115f609544f687aed070f1a61f0e, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/24cda499b21d47b59350bcd5728a426c, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/6c1ff151010b46c4a3dd84812f4069b4, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/6cf94b918db84e4da8056344fdb7def6, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/803c0ebb03de497597a4fb2603433b4c, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/dc730c770c004e8ea20bb161b82e625d, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/99515b6745ed440f80a0ad2bf9f8481c, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/aa4267b177a14e5998333550f082d8a6, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/5780b690fd37459ea70ebb514b42b34a, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/b2bdb591e16e427dbe6db6674ffeae25, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/aae955ad339f4e63b09c8ac9863f761a, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/8a288ec7dcee4b68b0dcb6153e0dc7f2, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/642acdba67f84af2b20066723551fa47, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/1134d581aef84f2f8bd11b732ccbe78b, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/107c4678604a4727bc1760700b46ae38, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/7a8f17bfe827445f96297fa297e562a8, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/c67a737510e5471db76bc689782246c5, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/ea344a8b54504ac6bef67542c932e792, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/edab9cc0111f4945a386f1573d1efb40, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/37384b7eed344d19ae3c21ebeb4c7823, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/77fe3a3d63c64de580f0330b14be5846, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/6d076088d3e745a0aa8405a71460d178, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/ead3ca3fb60949cdbedf2441bc0acf93, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/afb9c740adc7457083daeb07819b3a27, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/2d948b679bd44a318a5eb0b0558b9384, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/77e3e05f931241509e187715a572dc97, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/36d9863e74c44fd9858afafe8a72e980, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/cd551bfe2ff348fc82480844543cae6e, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/11f9a24bd9ea4268881aaf2d5e046a6b, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/60a60b1639414c1da097244c75f7273e, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/211b68e577744284a3580da1d4f6ce35, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/55cb2a0647d2491984eed76d4ae1d8d4, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/25195452765548deb73288a8e93ec5a5, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/1b13c1ead2d843fbac2fffba75172f78, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/3f76e3d7466a46d2b294c1a705583f3f, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/5a3cb07824f14ef2b802a9072d6db8ac, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/bb149c5e0e5d416ea0cce34b5b956223] to archive 2023-04-12 15:56:31,890 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-12 15:56:31,891 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/c3592a2ea53c4b34b78fbc83be19ebe2 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/c3592a2ea53c4b34b78fbc83be19ebe2 2023-04-12 15:56:31,892 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/14632b72474342a197ebfcab156ba9b6 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/14632b72474342a197ebfcab156ba9b6 2023-04-12 15:56:31,893 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/648406cb77ba4910b5b3ca50c90ceffe to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/648406cb77ba4910b5b3ca50c90ceffe 2023-04-12 15:56:31,894 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/58e51f98e0724a8b8acf6d815d7e9d6b to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/58e51f98e0724a8b8acf6d815d7e9d6b 2023-04-12 15:56:31,895 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/6f51d7e5811b4de6977036482e15daea to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/6f51d7e5811b4de6977036482e15daea 2023-04-12 15:56:31,895 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/e4d5a161e3b24a94a83578a20f1c850c to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/e4d5a161e3b24a94a83578a20f1c850c 2023-04-12 15:56:31,896 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/4c9ec657b3454f169fc44806f6254a70 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/4c9ec657b3454f169fc44806f6254a70 2023-04-12 15:56:31,897 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/071fc3308c3b41daa6b97b0e277421f0 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/071fc3308c3b41daa6b97b0e277421f0 2023-04-12 15:56:31,898 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/6d5d115f609544f687aed070f1a61f0e to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/6d5d115f609544f687aed070f1a61f0e 2023-04-12 15:56:31,898 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/24cda499b21d47b59350bcd5728a426c to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/24cda499b21d47b59350bcd5728a426c 2023-04-12 15:56:31,899 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/6c1ff151010b46c4a3dd84812f4069b4 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/6c1ff151010b46c4a3dd84812f4069b4 2023-04-12 15:56:31,900 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/6cf94b918db84e4da8056344fdb7def6 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/6cf94b918db84e4da8056344fdb7def6 2023-04-12 15:56:31,901 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/803c0ebb03de497597a4fb2603433b4c to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/803c0ebb03de497597a4fb2603433b4c 2023-04-12 15:56:31,901 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/dc730c770c004e8ea20bb161b82e625d to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/dc730c770c004e8ea20bb161b82e625d 2023-04-12 15:56:31,902 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/99515b6745ed440f80a0ad2bf9f8481c to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/99515b6745ed440f80a0ad2bf9f8481c 2023-04-12 15:56:31,903 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/aa4267b177a14e5998333550f082d8a6 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/aa4267b177a14e5998333550f082d8a6 2023-04-12 15:56:31,904 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/5780b690fd37459ea70ebb514b42b34a to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/5780b690fd37459ea70ebb514b42b34a 2023-04-12 15:56:31,904 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/b2bdb591e16e427dbe6db6674ffeae25 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/b2bdb591e16e427dbe6db6674ffeae25 2023-04-12 15:56:31,905 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/aae955ad339f4e63b09c8ac9863f761a to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/aae955ad339f4e63b09c8ac9863f761a 2023-04-12 15:56:31,906 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/8a288ec7dcee4b68b0dcb6153e0dc7f2 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/8a288ec7dcee4b68b0dcb6153e0dc7f2 2023-04-12 15:56:31,907 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/642acdba67f84af2b20066723551fa47 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/642acdba67f84af2b20066723551fa47 2023-04-12 15:56:31,907 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/1134d581aef84f2f8bd11b732ccbe78b to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/1134d581aef84f2f8bd11b732ccbe78b 2023-04-12 15:56:31,908 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/107c4678604a4727bc1760700b46ae38 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/107c4678604a4727bc1760700b46ae38 2023-04-12 15:56:31,909 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/7a8f17bfe827445f96297fa297e562a8 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/7a8f17bfe827445f96297fa297e562a8 2023-04-12 15:56:31,910 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/c67a737510e5471db76bc689782246c5 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/c67a737510e5471db76bc689782246c5 2023-04-12 15:56:31,910 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/ea344a8b54504ac6bef67542c932e792 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/ea344a8b54504ac6bef67542c932e792 2023-04-12 15:56:31,911 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/edab9cc0111f4945a386f1573d1efb40 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/edab9cc0111f4945a386f1573d1efb40 2023-04-12 15:56:31,912 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/37384b7eed344d19ae3c21ebeb4c7823 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/37384b7eed344d19ae3c21ebeb4c7823 2023-04-12 15:56:31,913 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/77fe3a3d63c64de580f0330b14be5846 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/77fe3a3d63c64de580f0330b14be5846 2023-04-12 15:56:31,914 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/6d076088d3e745a0aa8405a71460d178 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/6d076088d3e745a0aa8405a71460d178 2023-04-12 15:56:31,915 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/ead3ca3fb60949cdbedf2441bc0acf93 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/ead3ca3fb60949cdbedf2441bc0acf93 2023-04-12 15:56:31,915 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/afb9c740adc7457083daeb07819b3a27 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/afb9c740adc7457083daeb07819b3a27 2023-04-12 15:56:31,916 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/2d948b679bd44a318a5eb0b0558b9384 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/2d948b679bd44a318a5eb0b0558b9384 2023-04-12 15:56:31,917 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/77e3e05f931241509e187715a572dc97 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/77e3e05f931241509e187715a572dc97 2023-04-12 15:56:31,918 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/36d9863e74c44fd9858afafe8a72e980 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/36d9863e74c44fd9858afafe8a72e980 2023-04-12 15:56:31,919 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/cd551bfe2ff348fc82480844543cae6e to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/cd551bfe2ff348fc82480844543cae6e 2023-04-12 15:56:31,920 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/11f9a24bd9ea4268881aaf2d5e046a6b to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/11f9a24bd9ea4268881aaf2d5e046a6b 2023-04-12 15:56:31,921 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/60a60b1639414c1da097244c75f7273e to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/60a60b1639414c1da097244c75f7273e 2023-04-12 15:56:31,922 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/211b68e577744284a3580da1d4f6ce35 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/211b68e577744284a3580da1d4f6ce35 2023-04-12 15:56:31,923 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/55cb2a0647d2491984eed76d4ae1d8d4 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/55cb2a0647d2491984eed76d4ae1d8d4 2023-04-12 15:56:31,924 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/25195452765548deb73288a8e93ec5a5 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/25195452765548deb73288a8e93ec5a5 2023-04-12 15:56:31,924 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/1b13c1ead2d843fbac2fffba75172f78 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/1b13c1ead2d843fbac2fffba75172f78 2023-04-12 15:56:31,925 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/3f76e3d7466a46d2b294c1a705583f3f to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/3f76e3d7466a46d2b294c1a705583f3f 2023-04-12 15:56:31,926 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/5a3cb07824f14ef2b802a9072d6db8ac to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/5a3cb07824f14ef2b802a9072d6db8ac 2023-04-12 15:56:31,927 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/bb149c5e0e5d416ea0cce34b5b956223 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/bb149c5e0e5d416ea0cce34b5b956223 2023-04-12 15:56:31,930 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/9c4128112654460982a183f69f08ee69, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/8a45af98195842cca6ac0d6a2904aea9, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/3cb02f3b11254f98bdf01e3f1ca8ec56, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/2d27599b9e95413aa6532d8109489593, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/8a1856f32e9e41e0903d31b684291e2e, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/e99449b3fbf6449e8c82abdd7a292380, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/034bca029e534ba2b82e09fb77f6f21a, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/0390a91420364d879a552226718d1d03, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/e00661d35ae74bf681fe4cbd8757bddf, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/10f0ccd9f5504b98b8977fec54d82d72, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/b01faf10215543e89ab551370b386ccc, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/93f7f06f96324b8c9d4d5a49df0c99b9, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/70215b26de4541da8cd467d0489e6a9b, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/f0886de0f31943689c0fc9154a0922d2, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/e3bc615cc2ad43d1aeabd0924dbfc509, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/afc67f0e441448358a3612f3d25e97ad, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/6acefc2ba0b5407eacdcf759ecde9c21, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/3bd9468b1233439ea95cd0f943c05f5f, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/a89230cf08c442de9c60a355b7341823, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/15ed22be7f464fa59581fe57b17f67ec, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/3745e854d90d484dbbdd945b3a0f36fb, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/d260f3cb759f418ea7cfb00737f0d53c, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/d13099c165374393873d1dc2ca8dda2e, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/1734884edf2445bc8dac36d4f297a93a, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/b422434fd7204aaa959242621aed1a9c, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/faadc296ed7b45e5ba228f8d8eb77252, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/5b8d99a4d869480aad17d6a782c8a504, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/3997ea0586fc4e699ffc15825b05c9fd, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/e33cb229559b46f78c7a1a99f691ab84, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/85a304b6d02540129fced3ed0e5b71dc, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/58a1617d011c4349931d177145ff64b1, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/12631e3629864a439f00dd6e61aadb44, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/474092d4fe3142ec986b7887effde366, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/11c2c117cc2e481a8f06292d81e491e7, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/6e5f54175c004e44bb5e5ee928cdf2c4, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/e67caf640b0545e1a783f81b3655d89a, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/4b016300f9b44297a4e8254f7b738dc4, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/139a72300c38492c9fb4159827dea1c6, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/8da03c5f051c411198dde322a4b8a86a, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/d429a62d40b34216ba046da40ebffff0, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/00fedc1ec4bf48029bdc2b02b8171fd9, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/d2cfeaa3102746e895b06bc88df73d5a, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/2e4048381027401e8a9726c6bce77086, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/a082e532720e44a6926ed542fc275287, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/18c974752f8f4157a7c01ff25d3586ec, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/1962da4b34644b0999b641f120261a92] to archive 2023-04-12 15:56:31,931 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-12 15:56:31,932 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/9c4128112654460982a183f69f08ee69 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/9c4128112654460982a183f69f08ee69 2023-04-12 15:56:31,933 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/8a45af98195842cca6ac0d6a2904aea9 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/8a45af98195842cca6ac0d6a2904aea9 2023-04-12 15:56:31,934 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/3cb02f3b11254f98bdf01e3f1ca8ec56 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/3cb02f3b11254f98bdf01e3f1ca8ec56 2023-04-12 15:56:31,935 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/2d27599b9e95413aa6532d8109489593 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/2d27599b9e95413aa6532d8109489593 2023-04-12 15:56:31,935 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/8a1856f32e9e41e0903d31b684291e2e to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/8a1856f32e9e41e0903d31b684291e2e 2023-04-12 15:56:31,936 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/e99449b3fbf6449e8c82abdd7a292380 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/e99449b3fbf6449e8c82abdd7a292380 2023-04-12 15:56:31,937 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/034bca029e534ba2b82e09fb77f6f21a to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/034bca029e534ba2b82e09fb77f6f21a 2023-04-12 15:56:31,938 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/0390a91420364d879a552226718d1d03 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/0390a91420364d879a552226718d1d03 2023-04-12 15:56:31,938 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/e00661d35ae74bf681fe4cbd8757bddf to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/e00661d35ae74bf681fe4cbd8757bddf 2023-04-12 15:56:31,939 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/10f0ccd9f5504b98b8977fec54d82d72 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/10f0ccd9f5504b98b8977fec54d82d72 2023-04-12 15:56:31,940 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/b01faf10215543e89ab551370b386ccc to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/b01faf10215543e89ab551370b386ccc 2023-04-12 15:56:31,941 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/93f7f06f96324b8c9d4d5a49df0c99b9 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/93f7f06f96324b8c9d4d5a49df0c99b9 2023-04-12 15:56:31,942 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/70215b26de4541da8cd467d0489e6a9b to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/70215b26de4541da8cd467d0489e6a9b 2023-04-12 15:56:31,943 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/f0886de0f31943689c0fc9154a0922d2 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/f0886de0f31943689c0fc9154a0922d2 2023-04-12 15:56:31,943 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/e3bc615cc2ad43d1aeabd0924dbfc509 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/e3bc615cc2ad43d1aeabd0924dbfc509 2023-04-12 15:56:31,944 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/afc67f0e441448358a3612f3d25e97ad to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/afc67f0e441448358a3612f3d25e97ad 2023-04-12 15:56:31,945 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/6acefc2ba0b5407eacdcf759ecde9c21 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/6acefc2ba0b5407eacdcf759ecde9c21 2023-04-12 15:56:31,946 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/3bd9468b1233439ea95cd0f943c05f5f to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/3bd9468b1233439ea95cd0f943c05f5f 2023-04-12 15:56:31,947 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/a89230cf08c442de9c60a355b7341823 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/a89230cf08c442de9c60a355b7341823 2023-04-12 15:56:31,947 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/15ed22be7f464fa59581fe57b17f67ec to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/15ed22be7f464fa59581fe57b17f67ec 2023-04-12 15:56:31,948 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/3745e854d90d484dbbdd945b3a0f36fb to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/3745e854d90d484dbbdd945b3a0f36fb 2023-04-12 15:56:31,949 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/d260f3cb759f418ea7cfb00737f0d53c to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/d260f3cb759f418ea7cfb00737f0d53c 2023-04-12 15:56:31,950 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/d13099c165374393873d1dc2ca8dda2e to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/d13099c165374393873d1dc2ca8dda2e 2023-04-12 15:56:31,951 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/1734884edf2445bc8dac36d4f297a93a to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/1734884edf2445bc8dac36d4f297a93a 2023-04-12 15:56:31,951 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/b422434fd7204aaa959242621aed1a9c to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/b422434fd7204aaa959242621aed1a9c 2023-04-12 15:56:31,952 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/faadc296ed7b45e5ba228f8d8eb77252 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/faadc296ed7b45e5ba228f8d8eb77252 2023-04-12 15:56:31,953 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/5b8d99a4d869480aad17d6a782c8a504 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/5b8d99a4d869480aad17d6a782c8a504 2023-04-12 15:56:31,954 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/3997ea0586fc4e699ffc15825b05c9fd to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/3997ea0586fc4e699ffc15825b05c9fd 2023-04-12 15:56:31,955 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/e33cb229559b46f78c7a1a99f691ab84 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/e33cb229559b46f78c7a1a99f691ab84 2023-04-12 15:56:31,956 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/85a304b6d02540129fced3ed0e5b71dc to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/85a304b6d02540129fced3ed0e5b71dc 2023-04-12 15:56:31,956 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/58a1617d011c4349931d177145ff64b1 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/58a1617d011c4349931d177145ff64b1 2023-04-12 15:56:31,957 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/12631e3629864a439f00dd6e61aadb44 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/12631e3629864a439f00dd6e61aadb44 2023-04-12 15:56:31,958 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/474092d4fe3142ec986b7887effde366 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/474092d4fe3142ec986b7887effde366 2023-04-12 15:56:31,959 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/11c2c117cc2e481a8f06292d81e491e7 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/11c2c117cc2e481a8f06292d81e491e7 2023-04-12 15:56:31,959 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/6e5f54175c004e44bb5e5ee928cdf2c4 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/6e5f54175c004e44bb5e5ee928cdf2c4 2023-04-12 15:56:31,960 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/e67caf640b0545e1a783f81b3655d89a to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/e67caf640b0545e1a783f81b3655d89a 2023-04-12 15:56:31,961 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/4b016300f9b44297a4e8254f7b738dc4 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/4b016300f9b44297a4e8254f7b738dc4 2023-04-12 15:56:31,962 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/139a72300c38492c9fb4159827dea1c6 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/139a72300c38492c9fb4159827dea1c6 2023-04-12 15:56:31,963 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/8da03c5f051c411198dde322a4b8a86a to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/8da03c5f051c411198dde322a4b8a86a 2023-04-12 15:56:31,963 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/d429a62d40b34216ba046da40ebffff0 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/d429a62d40b34216ba046da40ebffff0 2023-04-12 15:56:31,964 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/00fedc1ec4bf48029bdc2b02b8171fd9 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/00fedc1ec4bf48029bdc2b02b8171fd9 2023-04-12 15:56:31,965 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/d2cfeaa3102746e895b06bc88df73d5a to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/d2cfeaa3102746e895b06bc88df73d5a 2023-04-12 15:56:31,966 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/2e4048381027401e8a9726c6bce77086 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/2e4048381027401e8a9726c6bce77086 2023-04-12 15:56:31,967 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/a082e532720e44a6926ed542fc275287 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/a082e532720e44a6926ed542fc275287 2023-04-12 15:56:31,967 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/18c974752f8f4157a7c01ff25d3586ec to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/18c974752f8f4157a7c01ff25d3586ec 2023-04-12 15:56:31,968 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/1962da4b34644b0999b641f120261a92 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/1962da4b34644b0999b641f120261a92 2023-04-12 15:56:31,971 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/53eb3046b978463496b92e9046a48f23, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/e787be0dd18748b398a7f7864163748d, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/a510f061d847430988389eae267a0dd8, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/01b75d793f774d7bba66c689e6c002b6, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/10e7fe83908e49d08e5fa67d9d15847a, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/50e88dcadbd247818ceeaa717c4a5250, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/7eafe023391147ebbb60b87401bd5e2e, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/c0567359de4546919d4c5915d62b0367, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/edd973233a354820888485ea48d455e1, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/50d1c61eb6c64c4b9801418dd9d5dc5a, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/d76c5cda3a31405997aa8220567caf75, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/31e495d15d5b4f41b259a39cb5581962, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/96a6307aae664f5a9d1863cdbddd6080, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/dc8c7268701c45fc8129e59e4aa97585, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/9822f06bd891491786541d15b386f2d0, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/45072bbdd9f5454a9dafd915d4100f68, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/9e5470faaf4942bda2fd94afe540f099, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/d848244d3cee450799a65fbdae5ac99f, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/b3c93ab9cbc3460cb52f0908ee68cfd2, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/45ecf715677543ecb84bc04fd8ffe1bb, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/8c1108d3fcde40528b3ce3526089f927, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/8ef1107d19c94ac391510d308aa5589e, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/f6344aa7a3514ab7b675fa186c0a6a91, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/66fcdac5789c40709a9317d4e01d4c3b, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/b200901720474088a08fef2b53adad2c, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/1453d62d6ea34aef800f08e3a61292a9, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/1e75af02aefe4d5a9500e8be10838bfe, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/daed925ea716444a9fd864c060de4178, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/7a8cc3dc8d32438bb6d343e749aa3144, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/46593273575e4d59b310d87326d7b0dd, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/0b9fdc14aec64f7e8ada8ad06eb674aa, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/5218664995954f6a97811f3158dc1bb7, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/924a62afc9264e7eabb1b8d9ba76f108, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/df3761fa0d3c4398bc41dd3ff9626f86, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/663db4c933884290a21af172f7a34a3a, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/268d259a4b9042f096586441a2cb9785, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/81a3c88580514d57a41154eaa484c309, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/0d37ec0e606942b28ed9852568edc41e, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/5a07130d038d402e8777cf9df6fe0a8f, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/cd1b04703da1480aa5511a963382e75d, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/ccac69043df34da584a00adbdafc3f9d, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/9d2ea6db7209483cb4030d2d88de66b2, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/d778e6f30ec744c3ae2fa9ec58771358, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/804b2e48cf6e483baa6276c67d076986, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/b00a6f42dbbf4f7f97df195a8fa04339] to archive 2023-04-12 15:56:31,972 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-12 15:56:31,973 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/53eb3046b978463496b92e9046a48f23 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/53eb3046b978463496b92e9046a48f23 2023-04-12 15:56:31,974 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/e787be0dd18748b398a7f7864163748d to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/e787be0dd18748b398a7f7864163748d 2023-04-12 15:56:31,975 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/a510f061d847430988389eae267a0dd8 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/a510f061d847430988389eae267a0dd8 2023-04-12 15:56:31,976 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/01b75d793f774d7bba66c689e6c002b6 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/01b75d793f774d7bba66c689e6c002b6 2023-04-12 15:56:31,976 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/10e7fe83908e49d08e5fa67d9d15847a to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/10e7fe83908e49d08e5fa67d9d15847a 2023-04-12 15:56:31,977 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/50e88dcadbd247818ceeaa717c4a5250 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/50e88dcadbd247818ceeaa717c4a5250 2023-04-12 15:56:31,978 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/7eafe023391147ebbb60b87401bd5e2e to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/7eafe023391147ebbb60b87401bd5e2e 2023-04-12 15:56:31,979 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/c0567359de4546919d4c5915d62b0367 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/c0567359de4546919d4c5915d62b0367 2023-04-12 15:56:31,979 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/edd973233a354820888485ea48d455e1 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/edd973233a354820888485ea48d455e1 2023-04-12 15:56:31,980 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/50d1c61eb6c64c4b9801418dd9d5dc5a to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/50d1c61eb6c64c4b9801418dd9d5dc5a 2023-04-12 15:56:31,981 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/d76c5cda3a31405997aa8220567caf75 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/d76c5cda3a31405997aa8220567caf75 2023-04-12 15:56:31,982 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/31e495d15d5b4f41b259a39cb5581962 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/31e495d15d5b4f41b259a39cb5581962 2023-04-12 15:56:31,983 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/96a6307aae664f5a9d1863cdbddd6080 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/96a6307aae664f5a9d1863cdbddd6080 2023-04-12 15:56:31,984 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/dc8c7268701c45fc8129e59e4aa97585 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/dc8c7268701c45fc8129e59e4aa97585 2023-04-12 15:56:31,984 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/9822f06bd891491786541d15b386f2d0 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/9822f06bd891491786541d15b386f2d0 2023-04-12 15:56:31,985 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/45072bbdd9f5454a9dafd915d4100f68 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/45072bbdd9f5454a9dafd915d4100f68 2023-04-12 15:56:31,986 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/9e5470faaf4942bda2fd94afe540f099 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/9e5470faaf4942bda2fd94afe540f099 2023-04-12 15:56:31,987 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/d848244d3cee450799a65fbdae5ac99f to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/d848244d3cee450799a65fbdae5ac99f 2023-04-12 15:56:31,988 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/b3c93ab9cbc3460cb52f0908ee68cfd2 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/b3c93ab9cbc3460cb52f0908ee68cfd2 2023-04-12 15:56:31,988 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/45ecf715677543ecb84bc04fd8ffe1bb to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/45ecf715677543ecb84bc04fd8ffe1bb 2023-04-12 15:56:31,989 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/8c1108d3fcde40528b3ce3526089f927 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/8c1108d3fcde40528b3ce3526089f927 2023-04-12 15:56:31,990 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/8ef1107d19c94ac391510d308aa5589e to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/8ef1107d19c94ac391510d308aa5589e 2023-04-12 15:56:31,991 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/f6344aa7a3514ab7b675fa186c0a6a91 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/f6344aa7a3514ab7b675fa186c0a6a91 2023-04-12 15:56:31,992 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/66fcdac5789c40709a9317d4e01d4c3b to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/66fcdac5789c40709a9317d4e01d4c3b 2023-04-12 15:56:31,993 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/b200901720474088a08fef2b53adad2c to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/b200901720474088a08fef2b53adad2c 2023-04-12 15:56:31,994 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/1453d62d6ea34aef800f08e3a61292a9 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/1453d62d6ea34aef800f08e3a61292a9 2023-04-12 15:56:31,994 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/1e75af02aefe4d5a9500e8be10838bfe to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/1e75af02aefe4d5a9500e8be10838bfe 2023-04-12 15:56:31,995 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/daed925ea716444a9fd864c060de4178 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/daed925ea716444a9fd864c060de4178 2023-04-12 15:56:31,996 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/7a8cc3dc8d32438bb6d343e749aa3144 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/7a8cc3dc8d32438bb6d343e749aa3144 2023-04-12 15:56:31,997 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/46593273575e4d59b310d87326d7b0dd to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/46593273575e4d59b310d87326d7b0dd 2023-04-12 15:56:31,998 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/0b9fdc14aec64f7e8ada8ad06eb674aa to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/0b9fdc14aec64f7e8ada8ad06eb674aa 2023-04-12 15:56:31,999 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/5218664995954f6a97811f3158dc1bb7 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/5218664995954f6a97811f3158dc1bb7 2023-04-12 15:56:31,999 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/924a62afc9264e7eabb1b8d9ba76f108 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/924a62afc9264e7eabb1b8d9ba76f108 2023-04-12 15:56:32,000 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/df3761fa0d3c4398bc41dd3ff9626f86 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/df3761fa0d3c4398bc41dd3ff9626f86 2023-04-12 15:56:32,001 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/663db4c933884290a21af172f7a34a3a to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/663db4c933884290a21af172f7a34a3a 2023-04-12 15:56:32,002 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/268d259a4b9042f096586441a2cb9785 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/268d259a4b9042f096586441a2cb9785 2023-04-12 15:56:32,004 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/81a3c88580514d57a41154eaa484c309 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/81a3c88580514d57a41154eaa484c309 2023-04-12 15:56:32,004 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/0d37ec0e606942b28ed9852568edc41e to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/0d37ec0e606942b28ed9852568edc41e 2023-04-12 15:56:32,005 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/5a07130d038d402e8777cf9df6fe0a8f to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/5a07130d038d402e8777cf9df6fe0a8f 2023-04-12 15:56:32,006 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/cd1b04703da1480aa5511a963382e75d to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/cd1b04703da1480aa5511a963382e75d 2023-04-12 15:56:32,007 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/ccac69043df34da584a00adbdafc3f9d to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/ccac69043df34da584a00adbdafc3f9d 2023-04-12 15:56:32,007 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/9d2ea6db7209483cb4030d2d88de66b2 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/9d2ea6db7209483cb4030d2d88de66b2 2023-04-12 15:56:32,008 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/d778e6f30ec744c3ae2fa9ec58771358 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/d778e6f30ec744c3ae2fa9ec58771358 2023-04-12 15:56:32,009 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/804b2e48cf6e483baa6276c67d076986 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/804b2e48cf6e483baa6276c67d076986 2023-04-12 15:56:32,010 DEBUG [StoreCloser-TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/b00a6f42dbbf4f7f97df195a8fa04339 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/b00a6f42dbbf4f7f97df195a8fa04339 2023-04-12 15:56:32,013 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/recovered.edits/1241.seqid, newMaxSeqId=1241, maxSeqId=4 2023-04-12 15:56:32,014 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03. 2023-04-12 15:56:32,014 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 4870003209f366bc50ef2084e6e2cf03: 2023-04-12 15:56:32,015 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 4870003209f366bc50ef2084e6e2cf03 2023-04-12 15:56:32,015 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=4870003209f366bc50ef2084e6e2cf03, regionState=CLOSED 2023-04-12 15:56:32,016 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681314992015"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681314992015"}]},"ts":"1681314992015"} 2023-04-12 15:56:32,017 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=64, resume processing ppid=63 2023-04-12 15:56:32,017 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=64, ppid=63, state=SUCCESS; CloseRegionProcedure 4870003209f366bc50ef2084e6e2cf03, server=jenkins-hbase4.apache.org,38905,1681314818739 in 1.7630 sec 2023-04-12 15:56:32,018 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=63, resume processing ppid=62 2023-04-12 15:56:32,018 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=63, ppid=62, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=4870003209f366bc50ef2084e6e2cf03, UNASSIGN in 1.7670 sec 2023-04-12 15:56:32,019 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681314992019"}]},"ts":"1681314992019"} 2023-04-12 15:56:32,019 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-12 15:56:32,021 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-12 15:56:32,022 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=62, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.7760 sec 2023-04-12 15:56:32,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-12 15:56:32,352 INFO [Listener at localhost/43583] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 62 completed 2023-04-12 15:56:32,352 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-12 15:56:32,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] procedure2.ProcedureExecutor(1029): Stored pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-12 15:56:32,354 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-12 15:56:32,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-12 15:56:32,355 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=65, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-12 15:56:32,360 DEBUG [HFileArchiver-13] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03 2023-04-12 15:56:32,361 DEBUG [HFileArchiver-13] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A, FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B, FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C, FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/recovered.edits] 2023-04-12 15:56:32,363 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/7fa3101e18534606bc52c046289413dc to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/7fa3101e18534606bc52c046289413dc 2023-04-12 15:56:32,364 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/f45beb74c52c4f72946210d2591e2400 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/A/f45beb74c52c4f72946210d2591e2400 2023-04-12 15:56:32,366 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/45a607c53ac14188b1ef9b88df719aed to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/45a607c53ac14188b1ef9b88df719aed 2023-04-12 15:56:32,367 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/7831d6052c1d4a1a873d6a9fad2ae4f9 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/B/7831d6052c1d4a1a873d6a9fad2ae4f9 2023-04-12 15:56:32,369 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/6a25a77417af41919d48e25d7af3d7b4 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/6a25a77417af41919d48e25d7af3d7b4 2023-04-12 15:56:32,370 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/d1b650c782ee4b47b269a5ec775cd446 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/C/d1b650c782ee4b47b269a5ec775cd446 2023-04-12 15:56:32,372 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/recovered.edits/1241.seqid to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03/recovered.edits/1241.seqid 2023-04-12 15:56:32,373 DEBUG [HFileArchiver-13] backup.HFileArchiver(596): Deleted hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/.tmp/data/default/TestAcidGuarantees/4870003209f366bc50ef2084e6e2cf03 2023-04-12 15:56:32,373 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-12 15:56:32,373 DEBUG [PEWorker-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-12 15:56:32,374 DEBUG [PEWorker-4] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-12 15:56:32,376 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041200576d01df0e48978bb6fa7e1302d2ce to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041200576d01df0e48978bb6fa7e1302d2ce 2023-04-12 15:56:32,377 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041203499a3ac93946708751cdf0a419ccd7 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041203499a3ac93946708751cdf0a419ccd7 2023-04-12 15:56:32,378 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304120fdd289d5907412188710244e845b64c to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304120fdd289d5907412188710244e845b64c 2023-04-12 15:56:32,378 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041210e553c7e3e443db85a73234b10ce7da to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041210e553c7e3e443db85a73234b10ce7da 2023-04-12 15:56:32,379 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304121af173b3e174448fba7b6a1ec217acdc to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304121af173b3e174448fba7b6a1ec217acdc 2023-04-12 15:56:32,380 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041227ea9efb2742478a8a5bd33564157674 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041227ea9efb2742478a8a5bd33564157674 2023-04-12 15:56:32,381 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304122c88d5c4ed214eaea4ce6d4c8d427cc0 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304122c88d5c4ed214eaea4ce6d4c8d427cc0 2023-04-12 15:56:32,382 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412304c09afebe945d0aa5fefd168b5ccc7 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412304c09afebe945d0aa5fefd168b5ccc7 2023-04-12 15:56:32,383 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304123d9f7b99d3304ff9a414ab308da6244e to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304123d9f7b99d3304ff9a414ab308da6244e 2023-04-12 15:56:32,384 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041241bb833ca39f49628a010066c7c0c880 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041241bb833ca39f49628a010066c7c0c880 2023-04-12 15:56:32,384 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412568da4c0ad98472ea4c30475ae85658d to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412568da4c0ad98472ea4c30475ae85658d 2023-04-12 15:56:32,385 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041256ae6ceaf48b4c94a9801898356c7bb4 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041256ae6ceaf48b4c94a9801898356c7bb4 2023-04-12 15:56:32,386 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304125bad526d58d94a8e8137b6c6ea22ad57 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304125bad526d58d94a8e8137b6c6ea22ad57 2023-04-12 15:56:32,387 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304125f7f4cbb7f184a0f854a8ace437430af to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304125f7f4cbb7f184a0f854a8ace437430af 2023-04-12 15:56:32,388 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041271256f9021ee4af6843ca689e34ae788 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041271256f9021ee4af6843ca689e34ae788 2023-04-12 15:56:32,388 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041276435b4e8dd248b4b0a8116a0f1ab1d1 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041276435b4e8dd248b4b0a8116a0f1ab1d1 2023-04-12 15:56:32,389 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412784e3113c20b4877b24d4f616dc0a7e1 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412784e3113c20b4877b24d4f616dc0a7e1 2023-04-12 15:56:32,390 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304127c804f500f494435ad4b3cc3996d2046 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304127c804f500f494435ad4b3cc3996d2046 2023-04-12 15:56:32,391 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304127e3ce1e0bded4442992c022d9ce06c4c to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304127e3ce1e0bded4442992c022d9ce06c4c 2023-04-12 15:56:32,392 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304127f2b3dcc8e39477387f1d2952809fb75 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304127f2b3dcc8e39477387f1d2952809fb75 2023-04-12 15:56:32,393 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304127f5e5b2ab661493e90bd6dfe1b7d0ccc to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304127f5e5b2ab661493e90bd6dfe1b7d0ccc 2023-04-12 15:56:32,394 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041280c9914cace340bf9df7fc71094c65ca to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041280c9914cace340bf9df7fc71094c65ca 2023-04-12 15:56:32,395 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041287deb95c1c504b42a8ff3215ae84aecf to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041287deb95c1c504b42a8ff3215ae84aecf 2023-04-12 15:56:32,396 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041291cf5c308ced41ecab7262cf301436e7 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041291cf5c308ced41ecab7262cf301436e7 2023-04-12 15:56:32,396 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412ac035439b4984e17b8e2da243883b2dd to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412ac035439b4984e17b8e2da243883b2dd 2023-04-12 15:56:32,397 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412ac9d7b1fb45a46b3a5e8be1276788c1f to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412ac9d7b1fb45a46b3a5e8be1276788c1f 2023-04-12 15:56:32,398 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412ba3d370591de4a8183dea2e2144292f3 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412ba3d370591de4a8183dea2e2144292f3 2023-04-12 15:56:32,399 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412ba60dbc29813438c8002eaabf1ad524e to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412ba60dbc29813438c8002eaabf1ad524e 2023-04-12 15:56:32,399 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412bd1060033d7a4df282b692afdd1a56b2 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412bd1060033d7a4df282b692afdd1a56b2 2023-04-12 15:56:32,400 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412c1d6542811424cf0ba96cdbfacb3fd84 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412c1d6542811424cf0ba96cdbfacb3fd84 2023-04-12 15:56:32,401 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412c31bed3ae125489e86fb0cc8325bb214 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412c31bed3ae125489e86fb0cc8325bb214 2023-04-12 15:56:32,402 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412d09eeced51bf4fa197d583085e635a1b to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412d09eeced51bf4fa197d583085e635a1b 2023-04-12 15:56:32,402 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412d18204bdaa4140dcb2691a8f1e13f5fe to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412d18204bdaa4140dcb2691a8f1e13f5fe 2023-04-12 15:56:32,403 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412d20195d3990a450586e36c41152da10a to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412d20195d3990a450586e36c41152da10a 2023-04-12 15:56:32,404 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412f9867d4be0804de7a0c5d07a6b34dbcc to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412f9867d4be0804de7a0c5d07a6b34dbcc 2023-04-12 15:56:32,405 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412fd2a3aa6dc5644d08758940050028903 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412fd2a3aa6dc5644d08758940050028903 2023-04-12 15:56:32,405 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412fdea57e19ae44beba16f68a41dc6d1b2 to hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412fdea57e19ae44beba16f68a41dc6d1b2 2023-04-12 15:56:32,406 DEBUG [PEWorker-4] backup.HFileArchiver(596): Deleted hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-12 15:56:32,407 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=65, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-12 15:56:32,409 WARN [PEWorker-4] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-12 15:56:32,410 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-12 15:56:32,411 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=65, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-12 15:56:32,411 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-12 15:56:32,411 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681314992411"}]},"ts":"9223372036854775807"} 2023-04-12 15:56:32,412 INFO [PEWorker-4] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-12 15:56:32,412 DEBUG [PEWorker-4] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 4870003209f366bc50ef2084e6e2cf03, NAME => 'TestAcidGuarantees,,1681314964379.4870003209f366bc50ef2084e6e2cf03.', STARTKEY => '', ENDKEY => ''}] 2023-04-12 15:56:32,412 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-12 15:56:32,412 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681314992412"}]},"ts":"9223372036854775807"} 2023-04-12 15:56:32,413 INFO [PEWorker-4] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-12 15:56:32,415 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(130): Finished pid=65, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-12 15:56:32,416 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=65, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 63 msec 2023-04-12 15:56:32,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36273] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-12 15:56:32,456 INFO [Listener at localhost/43583] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 65 completed 2023-04-12 15:56:32,467 INFO [Listener at localhost/43583] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=229 (was 232), OpenFileDescriptor=564 (was 576), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=409 (was 396) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=948 (was 860) - AvailableMemoryMB LEAK? - 2023-04-12 15:56:32,467 INFO [Listener at localhost/43583] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-04-12 15:56:32,467 INFO [Listener at localhost/43583] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-12 15:56:32,468 DEBUG [Listener at localhost/43583] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5e324982 to 127.0.0.1:60154 2023-04-12 15:56:32,468 DEBUG [Listener at localhost/43583] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 15:56:32,468 DEBUG [Listener at localhost/43583] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-04-12 15:56:32,469 DEBUG [Listener at localhost/43583] util.JVMClusterUtil(257): Found active master hash=195877632, stopped=false 2023-04-12 15:56:32,469 INFO [Listener at localhost/43583] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase4.apache.org,36273,1681314817519 2023-04-12 15:56:32,471 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-12 15:56:32,471 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-12 15:56:32,471 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-12 15:56:32,471 INFO [Listener at localhost/43583] procedure2.ProcedureExecutor(629): Stopping 2023-04-12 15:56:32,471 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-12 15:56:32,472 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-12 15:56:32,472 DEBUG [Listener at localhost/43583] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7afa6221 to 127.0.0.1:60154 2023-04-12 15:56:32,472 DEBUG [Listener at localhost/43583] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 15:56:32,472 INFO [Listener at localhost/43583] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,38905,1681314818739' ***** 2023-04-12 15:56:32,472 INFO [Listener at localhost/43583] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-04-12 15:56:32,472 INFO [RS:0;jenkins-hbase4:38905] regionserver.HeapMemoryManager(220): Stopping 2023-04-12 15:56:32,473 INFO [RS:0;jenkins-hbase4:38905] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-04-12 15:56:32,473 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-04-12 15:56:32,473 INFO [RS:0;jenkins-hbase4:38905] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-04-12 15:56:32,473 INFO [RS:0;jenkins-hbase4:38905] regionserver.HRegionServer(3304): Received CLOSE for 1b2138e70df333d56ab86f7247aa4167 2023-04-12 15:56:32,474 INFO [RS:0;jenkins-hbase4:38905] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:32,474 DEBUG [RS:0;jenkins-hbase4:38905] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0d7a4eb8 to 127.0.0.1:60154 2023-04-12 15:56:32,474 DEBUG [RS:0;jenkins-hbase4:38905] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 15:56:32,474 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1b2138e70df333d56ab86f7247aa4167, disabling compactions & flushes 2023-04-12 15:56:32,474 INFO [RS:0;jenkins-hbase4:38905] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-04-12 15:56:32,474 INFO [RS:0;jenkins-hbase4:38905] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-04-12 15:56:32,475 INFO [RS:0;jenkins-hbase4:38905] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-04-12 15:56:32,474 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1681314821437.1b2138e70df333d56ab86f7247aa4167. 2023-04-12 15:56:32,475 INFO [RS:0;jenkins-hbase4:38905] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-04-12 15:56:32,475 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1681314821437.1b2138e70df333d56ab86f7247aa4167. 2023-04-12 15:56:32,475 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1681314821437.1b2138e70df333d56ab86f7247aa4167. after waiting 0 ms 2023-04-12 15:56:32,475 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1681314821437.1b2138e70df333d56ab86f7247aa4167. 2023-04-12 15:56:32,475 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1b2138e70df333d56ab86f7247aa4167 1/1 column families, dataSize=78 B heapSize=488 B 2023-04-12 15:56:32,475 INFO [RS:0;jenkins-hbase4:38905] regionserver.HRegionServer(1475): Waiting on 2 regions to close 2023-04-12 15:56:32,475 DEBUG [RS:0;jenkins-hbase4:38905] regionserver.HRegionServer(1479): Online Regions={1588230740=hbase:meta,,1.1588230740, 1b2138e70df333d56ab86f7247aa4167=hbase:namespace,,1681314821437.1b2138e70df333d56ab86f7247aa4167.} 2023-04-12 15:56:32,475 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-04-12 15:56:32,475 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-12 15:56:32,475 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-12 15:56:32,476 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-12 15:56:32,476 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-04-12 15:56:32,476 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-12 15:56:32,476 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=20.92 KB heapSize=36.24 KB 2023-04-12 15:56:32,477 DEBUG [RS:0;jenkins-hbase4:38905] regionserver.HRegionServer(1505): Waiting on 1588230740, 1b2138e70df333d56ab86f7247aa4167 2023-04-12 15:56:32,677 DEBUG [RS:0;jenkins-hbase4:38905] regionserver.HRegionServer(1505): Waiting on 1588230740, 1b2138e70df333d56ab86f7247aa4167 2023-04-12 15:56:32,877 DEBUG [RS:0;jenkins-hbase4:38905] regionserver.HRegionServer(1505): Waiting on 1588230740, 1b2138e70df333d56ab86f7247aa4167 2023-04-12 15:56:32,883 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=18.26 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/hbase/meta/1588230740/.tmp/info/414cb5a02c954fb9a11fb62f4e18554d 2023-04-12 15:56:32,886 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/hbase/namespace/1b2138e70df333d56ab86f7247aa4167/.tmp/info/be2c8e67f8cd404c806f071243db3bd0 2023-04-12 15:56:32,888 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 414cb5a02c954fb9a11fb62f4e18554d 2023-04-12 15:56:32,890 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/hbase/namespace/1b2138e70df333d56ab86f7247aa4167/.tmp/info/be2c8e67f8cd404c806f071243db3bd0 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/hbase/namespace/1b2138e70df333d56ab86f7247aa4167/info/be2c8e67f8cd404c806f071243db3bd0 2023-04-12 15:56:32,893 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=588 B at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/hbase/meta/1588230740/.tmp/rep_barrier/3c19cf8cfd3347daaf5bb8f2e9f6367a 2023-04-12 15:56:32,894 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/hbase/namespace/1b2138e70df333d56ab86f7247aa4167/info/be2c8e67f8cd404c806f071243db3bd0, entries=2, sequenceid=6, filesize=4.8 K 2023-04-12 15:56:32,894 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 1b2138e70df333d56ab86f7247aa4167 in 419ms, sequenceid=6, compaction requested=false 2023-04-12 15:56:32,897 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 3c19cf8cfd3347daaf5bb8f2e9f6367a 2023-04-12 15:56:32,898 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/hbase/namespace/1b2138e70df333d56ab86f7247aa4167/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-04-12 15:56:32,901 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1681314821437.1b2138e70df333d56ab86f7247aa4167. 2023-04-12 15:56:32,901 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1b2138e70df333d56ab86f7247aa4167: 2023-04-12 15:56:32,901 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1681314821437.1b2138e70df333d56ab86f7247aa4167. 2023-04-12 15:56:32,907 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.08 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/hbase/meta/1588230740/.tmp/table/a7a57eda34a04487a0fa43d5f7a63fc3 2023-04-12 15:56:32,910 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for a7a57eda34a04487a0fa43d5f7a63fc3 2023-04-12 15:56:32,910 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/hbase/meta/1588230740/.tmp/info/414cb5a02c954fb9a11fb62f4e18554d as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/hbase/meta/1588230740/info/414cb5a02c954fb9a11fb62f4e18554d 2023-04-12 15:56:32,913 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 414cb5a02c954fb9a11fb62f4e18554d 2023-04-12 15:56:32,913 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/hbase/meta/1588230740/info/414cb5a02c954fb9a11fb62f4e18554d, entries=22, sequenceid=93, filesize=7.1 K 2023-04-12 15:56:32,914 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/hbase/meta/1588230740/.tmp/rep_barrier/3c19cf8cfd3347daaf5bb8f2e9f6367a as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/hbase/meta/1588230740/rep_barrier/3c19cf8cfd3347daaf5bb8f2e9f6367a 2023-04-12 15:56:32,916 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 3c19cf8cfd3347daaf5bb8f2e9f6367a 2023-04-12 15:56:32,916 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/hbase/meta/1588230740/rep_barrier/3c19cf8cfd3347daaf5bb8f2e9f6367a, entries=6, sequenceid=93, filesize=5.5 K 2023-04-12 15:56:32,917 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/hbase/meta/1588230740/.tmp/table/a7a57eda34a04487a0fa43d5f7a63fc3 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/hbase/meta/1588230740/table/a7a57eda34a04487a0fa43d5f7a63fc3 2023-04-12 15:56:32,919 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for a7a57eda34a04487a0fa43d5f7a63fc3 2023-04-12 15:56:32,919 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/hbase/meta/1588230740/table/a7a57eda34a04487a0fa43d5f7a63fc3, entries=9, sequenceid=93, filesize=5.5 K 2023-04-12 15:56:32,920 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.92 KB/21417, heapSize ~36.20 KB/37064, currentSize=0 B/0 for 1588230740 in 444ms, sequenceid=93, compaction requested=false 2023-04-12 15:56:32,923 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/data/hbase/meta/1588230740/recovered.edits/96.seqid, newMaxSeqId=96, maxSeqId=1 2023-04-12 15:56:32,924 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-12 15:56:32,925 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-12 15:56:32,925 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-12 15:56:32,925 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-04-12 15:56:33,027 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-04-12 15:56:33,027 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-04-12 15:56:33,078 INFO [RS:0;jenkins-hbase4:38905] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,38905,1681314818739; all regions closed. 2023-04-12 15:56:33,085 DEBUG [RS:0;jenkins-hbase4:38905] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/oldWALs 2023-04-12 15:56:33,085 INFO [RS:0;jenkins-hbase4:38905] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C38905%2C1681314818739.meta:.meta(num 1681314821179) 2023-04-12 15:56:33,089 DEBUG [RS:0;jenkins-hbase4:38905] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/oldWALs 2023-04-12 15:56:33,089 INFO [RS:0;jenkins-hbase4:38905] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C38905%2C1681314818739:(num 1681314820181) 2023-04-12 15:56:33,089 DEBUG [RS:0;jenkins-hbase4:38905] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 15:56:33,089 INFO [RS:0;jenkins-hbase4:38905] regionserver.LeaseManager(133): Closed leases 2023-04-12 15:56:33,090 INFO [RS:0;jenkins-hbase4:38905] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-04-12 15:56:33,090 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-12 15:56:33,091 INFO [RS:0;jenkins-hbase4:38905] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:38905 2023-04-12 15:56:33,096 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-12 15:56:33,096 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,38905,1681314818739 2023-04-12 15:56:33,096 ERROR [Listener at localhost/43583-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@1f26a563 rejected from java.util.concurrent.ThreadPoolExecutor@1b19325f[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2539] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-12 15:56:33,097 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-12 15:56:33,097 ERROR [Listener at localhost/43583-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@4528bd95 rejected from java.util.concurrent.ThreadPoolExecutor@1b19325f[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2539] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-12 15:56:33,098 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,38905,1681314818739] 2023-04-12 15:56:33,099 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,38905,1681314818739; numProcessing=1 2023-04-12 15:56:33,100 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,38905,1681314818739 already deleted, retry=false 2023-04-12 15:56:33,100 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,38905,1681314818739 expired; onlineServers=0 2023-04-12 15:56:33,100 INFO [RegionServerTracker-0] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,36273,1681314817519' ***** 2023-04-12 15:56:33,100 INFO [RegionServerTracker-0] regionserver.HRegionServer(2310): STOPPED: Cluster shutdown set; onlineServer=0 2023-04-12 15:56:33,100 DEBUG [M:0;jenkins-hbase4:36273] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4a91d13b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-04-12 15:56:33,100 INFO [M:0;jenkins-hbase4:36273] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,36273,1681314817519 2023-04-12 15:56:33,100 INFO [M:0;jenkins-hbase4:36273] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,36273,1681314817519; all regions closed. 2023-04-12 15:56:33,100 DEBUG [M:0;jenkins-hbase4:36273] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 15:56:33,100 DEBUG [M:0;jenkins-hbase4:36273] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-04-12 15:56:33,100 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-04-12 15:56:33,101 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1681314819890] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1681314819890,5,FailOnTimeoutGroup] 2023-04-12 15:56:33,101 DEBUG [M:0;jenkins-hbase4:36273] cleaner.HFileCleaner(317): Stopping file delete threads 2023-04-12 15:56:33,101 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1681314819889] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1681314819889,5,FailOnTimeoutGroup] 2023-04-12 15:56:33,101 INFO [M:0;jenkins-hbase4:36273] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-04-12 15:56:33,101 INFO [M:0;jenkins-hbase4:36273] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-04-12 15:56:33,101 INFO [M:0;jenkins-hbase4:36273] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-04-12 15:56:33,101 DEBUG [M:0;jenkins-hbase4:36273] master.HMaster(1512): Stopping service threads 2023-04-12 15:56:33,101 INFO [M:0;jenkins-hbase4:36273] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-04-12 15:56:33,102 ERROR [M:0;jenkins-hbase4:36273] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-11,5,PEWorkerGroup] Thread[HFileArchiver-12,5,PEWorkerGroup] Thread[IPC Client (1325659896) connection to localhost/127.0.0.1:40585 from jenkins,5,PEWorkerGroup] Thread[HFileArchiver-13,5,PEWorkerGroup] 2023-04-12 15:56:33,102 INFO [M:0;jenkins-hbase4:36273] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-12 15:56:33,102 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-04-12 15:56:33,105 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-04-12 15:56:33,105 DEBUG [M:0;jenkins-hbase4:36273] zookeeper.ZKUtil(398): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-04-12 15:56:33,105 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-12 15:56:33,106 WARN [M:0;jenkins-hbase4:36273] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-04-12 15:56:33,106 INFO [M:0;jenkins-hbase4:36273] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-12 15:56:33,106 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-12 15:56:33,106 INFO [M:0;jenkins-hbase4:36273] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-04-12 15:56:33,106 DEBUG [M:0;jenkins-hbase4:36273] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-12 15:56:33,106 INFO [M:0;jenkins-hbase4:36273] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-12 15:56:33,106 DEBUG [M:0;jenkins-hbase4:36273] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-12 15:56:33,106 DEBUG [M:0;jenkins-hbase4:36273] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-12 15:56:33,106 DEBUG [M:0;jenkins-hbase4:36273] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-12 15:56:33,106 INFO [M:0;jenkins-hbase4:36273] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=344.87 KB heapSize=393.67 KB 2023-04-12 15:56:33,199 INFO [RS:0;jenkins-hbase4:38905] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,38905,1681314818739; zookeeper connection closed. 2023-04-12 15:56:33,199 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-12 15:56:33,199 ERROR [Listener at localhost/43583-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@3d56598a rejected from java.util.concurrent.ThreadPoolExecutor@1b19325f[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2539] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-12 15:56:33,199 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@2f6a4f41] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@2f6a4f41 2023-04-12 15:56:33,200 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): regionserver:38905-0x10179140c5c0001, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-12 15:56:33,200 ERROR [Listener at localhost/43583-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@3e708651 rejected from java.util.concurrent.ThreadPoolExecutor@1b19325f[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2539] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-12 15:56:33,200 INFO [Listener at localhost/43583] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 1 regionserver(s) complete 2023-04-12 15:56:33,518 INFO [M:0;jenkins-hbase4:36273] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=344.87 KB at sequenceid=578 (bloomFilter=true), to=hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/7b919fa090024557994a533efbcf9cc0 2023-04-12 15:56:33,521 INFO [M:0;jenkins-hbase4:36273] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 7b919fa090024557994a533efbcf9cc0 2023-04-12 15:56:33,521 DEBUG [M:0;jenkins-hbase4:36273] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/7b919fa090024557994a533efbcf9cc0 as hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/7b919fa090024557994a533efbcf9cc0 2023-04-12 15:56:33,524 INFO [M:0;jenkins-hbase4:36273] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 7b919fa090024557994a533efbcf9cc0 2023-04-12 15:56:33,524 INFO [M:0;jenkins-hbase4:36273] regionserver.HStore(1080): Added hdfs://localhost:40585/user/jenkins/test-data/8488be88-31d8-7772-a68e-9d9803d13cb8/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/7b919fa090024557994a533efbcf9cc0, entries=65, sequenceid=578, filesize=24.7 K 2023-04-12 15:56:33,525 INFO [M:0;jenkins-hbase4:36273] regionserver.HRegion(2948): Finished flush of dataSize ~344.87 KB/353148, heapSize ~393.66 KB/403104, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 419ms, sequenceid=578, compaction requested=false 2023-04-12 15:56:33,525 INFO [M:0;jenkins-hbase4:36273] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-12 15:56:33,525 DEBUG [M:0;jenkins-hbase4:36273] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-12 15:56:33,528 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-12 15:56:33,528 INFO [M:0;jenkins-hbase4:36273] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-04-12 15:56:33,529 INFO [M:0;jenkins-hbase4:36273] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:36273 2023-04-12 15:56:33,530 DEBUG [M:0;jenkins-hbase4:36273] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,36273,1681314817519 already deleted, retry=false 2023-04-12 15:56:33,632 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-12 15:56:33,632 INFO [M:0;jenkins-hbase4:36273] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,36273,1681314817519; zookeeper connection closed. 2023-04-12 15:56:33,632 ERROR [Listener at localhost/43583-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@e57deaa rejected from java.util.concurrent.ThreadPoolExecutor@75a0b68f[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4244] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-12 15:56:33,632 DEBUG [Listener at localhost/43583-EventThread] zookeeper.ZKWatcher(600): master:36273-0x10179140c5c0000, quorum=127.0.0.1:60154, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-12 15:56:33,632 ERROR [Listener at localhost/43583-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@2e523c7c rejected from java.util.concurrent.ThreadPoolExecutor@75a0b68f[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4244] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-12 15:56:33,634 WARN [Listener at localhost/43583] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-04-12 15:56:33,639 INFO [Listener at localhost/43583] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-04-12 15:56:33,748 WARN [BP-1985789108-172.31.14.131-1681314814180 heartbeating to localhost/127.0.0.1:40585] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-04-12 15:56:33,748 WARN [BP-1985789108-172.31.14.131-1681314814180 heartbeating to localhost/127.0.0.1:40585] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1985789108-172.31.14.131-1681314814180 (Datanode Uuid 7b7d27e7-67e1-443a-9646-93169ca9f830) service to localhost/127.0.0.1:40585 2023-04-12 15:56:33,751 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a77bf0d8-cc1b-e9fb-fa24-4731623e8a8f/cluster_d313ccff-bce6-73f3-294c-14f839051cbd/dfs/data/data1/current/BP-1985789108-172.31.14.131-1681314814180] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-12 15:56:33,751 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a77bf0d8-cc1b-e9fb-fa24-4731623e8a8f/cluster_d313ccff-bce6-73f3-294c-14f839051cbd/dfs/data/data2/current/BP-1985789108-172.31.14.131-1681314814180] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-12 15:56:33,791 INFO [Listener at localhost/43583] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-04-12 15:56:33,982 INFO [Listener at localhost/43583] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-04-12 15:56:34,181 INFO [Listener at localhost/43583] hbase.HBaseTestingUtility(1293): Minicluster is down